Quantcast
Channel: DDL – Cloud Data Architect
Viewing all 275 articles
Browse latest View live

Understanding MySQL innodb_flush_log_at_trx_commit variable

$
0
0

Feed: Planet MySQL
;
Author: rathishDBA
;

The main objective of this post is to understand why innodb_fush_log_at_trx_commit variable is used? What are the applicable values for innodb_fush_log_at_trx_commit? How innodb_fush_log_at_trx_commit value will impact the MySQL performance and data safety?  How to change innodb_fush_log_at_trx_commit value? and How to change the frequency of InnoDB log flush?

Why innodb_fush_log_at_trx_commit?

In order to understand why we need innodb_fush_log_at_trx_commit variable, we should know about how InnoDB works. It is a huge and complex topic and it is not the scope of this article. I am trying to cover this topic in simple words and it is given below:
  1. InnoDB performs most of its operations at the memory (InnoDB Buffer Pool)
  2. It will write all changes from memory to the transaction log (InnoDB Log File)
  3. From transaction log – it will flush (write) data to storage disk (durable storage)
In order to achieve durability of data, we need to store each and every transaction data into hard disk storage. But consider, in a busy system, for each transaction commit, if InnoDB trying to flush (write) data to slow running disk, what will happen?, So how do we manage this situation, where we need to store each transaction data and at the same time maintaining good performance of the system.
The InnoDB provides the solution for this situation, based on your system, you can tell InnoDB, when you want to flush (write) data to disk. For example, you can tell InnoDB to work as mentioned below:
  1. Write to log file and flush data to disk in specified interval, not for every transaction commit.
  2. Write to log and flush to disk for each transaction commit.
  3. Write to log for every transaction commit but flush to disk at an interval not for each transaction commit.


Based on the application standard you can adjust these settings to maintain balance between performance and data safety. InnoDB provides a configurable variable to achieve this balance, this variable is called innodb_fush_log_at_trx_commit. This variable controls how often the log buffer is flushed.

What are the applicable values for innodb_fush_log_at_trx_commit and its impact?

InnoDB supports following 3 values for innodb_fush_log_at_trx_commit variable. Let see how these variables will change flush to disk behaviour:
innodb_flush_log_trx_at_commit=0
When you set innodb_flush_log_trx_at_commit=0, InnoDB will write the modified data (in InnoDB Buffer Pool) to log file (ib_logfile) and flush the log file (write to disk) every second, but it will not do anything at transaction commit.
Here, in case if there is a power failure or system crash, all the unflushed data will not be recoverable, since, it is not either written to log file or stored disk.
innodb_flush_log_trx_at_commit=1
When you set innodb_flush_log_trx_commit=1, InnoDB will write the log buffer to transaction log and flush to durable storage for every transaction.
Here, for all transaction commit, InnoDB will write to log and then write to disk, if in case the slower disk storage, it will badly impact the performance, i.e. the number of InnoDB transaction per second will be reduced.
innodb_flush_log_trx_at_commit=2
When you set innodb_flush_log_trx_commit = 2, InnoDB will write the log buffer to log file at every commit, but don’t write data to disk. InnoDB flushes data once in every second.
Option 2, even if there is a power failure or system crash, data will be available in log file and can be recoverable.


So which option to be used on your application?

We have seen, the behaviour of each values, now it is based on the application requirement, we need to choose this variable value.
If performance is the main goal, you can set the value as 2. Since, InnoDB writing to disk once in every second, not for every transaction commit and it will improve the performance dramatically. In case if there is power failure or crash, data can be recoverable from transaction log.
If data safety is the main goal, you can set the value as 1, so that for every transaction commit, InnoDB will flush to disk. But performance may reduce in this case.
If you set value as 0, InnoDB will write from buffer to log once in every second and it will not perform any flush to disk operation for every transaction commit. The problem with this option is if in case of any power failure or system crash, there may be a chance of losing data up to one second.

Key points:

During buffer to log operation, i.e. writing from InnoDB buffer pool to InnoDB transaction log file – data is simply moved from InnoDB buffer to Operating system’s cache, actually not written to the durable storage. So if you set innodb_fush_log_at_trx_commit either 0 or 2, there is a possibility of losing data up to one second.
If innodb_fush_log_at_trx_commit is set to 1 – InnoDB compels Operating System to flush data to durable storage. Writing to disk is a slow process and it is I/O blocking operation to ensure data is written to disk. So using this option, there is a chance; number of transaction per second will be reduced.
Note that, by default MySQL will auto commit the transactions.
Setting innodb_fush_log_at_trx_commit as 2 is useful when restoring huge databases, it will reduce the restoration time dramatically, and there are different opinions on this point, it is better to test in our own hand. In my experience, it really helped with the reduced restoration time.
Data Definition Language (DDL) changes flush the InnoDB log, independent of the innodb_fush_log_at_trx_commit setting.
Crash recovery operation works regardless of the innodb_fush_log_at_trx_commit setting, I am writing another article on InnoDB crash recovery.


How to configure innodb_fush_log_at_trx_commit variable?

The scope of the innodb_fush_log_at_trx_commit is GLOBAL and it can be set at dynamically without restarting server.
Dynamically on command line you can set innodb_fush_log_at_trx_commit as follows:
SET GLOBAL innodb_fush_log_at_trx_commit = 2;
On configuration file, you can set it as follows:

[mysqld]
innodb_fush_log_at_trx_commit = 2

Note: It requires server restart. Before making changes to configuration file, analyse the impact by setting it on dynamically.
If you try to set innodb_fush_log_at_trx_commit as session level variables, you will encounter the following error:

mysql> set session innodb_fush_log_at_trx_commit = 2;
ERROR 1229 (HY000): Variable ‘innodb_fush_log_at_trx_commit’ is a GLOBAL variable and should be set with SET GLOBAL
mysql>

How to change the frequency of InnoDB log file flushing?

The variable innodb_flush_log_at_timeout controls the frequency in which InnoDB flush log files. The frequency ranges from 1 to 2700 seconds, with the default value 1.


Note that, the higher this number, there is a higher the chance of losing data, in case of power failure or system crash.
For example, if you set this value as 5 seconds, in case of power failure, you may lose data upto 5 seconds.
In replication topology, to maintain durability and consistency of data, you can leave the default value, i.e. innodb_fush_log_at_trx_commit = 1.
I hope this article help you to understand, how InnoDB flush data and how it impact MySQL performance and data safety, also, how you can configure this variable to achieve maximum benefit. 
In future article let us see about how InnoDB crash recovery works and its configuration settings. If I missed something or if you wish to share your thoughts on this article, please mention in comment section, I will edit the post after review.


MySQL, Percona Server for MySQL and MariaDB Default Configuration Differences

$
0
0

Feed: Percona Database Performance Blog.
Author: Bradley Mickel.

 | October 9, 2017 | 
Posted In: InnoDB, Insight for DBAs, MariaDB, MySQL

PREVIOUS POST

MySQL and MariaDB Default ConfigurationIn this blog post, I’ll discuss some of the MySQL and MariaDB default configuration differences, focusing on MySQL 5.7 and MariaDB 10.2. Percona Server for MySQL uses the same defaults as MySQL, so I will not list them separately.

MariaDB Server is a general purpose open source database, created by the founders of MySQL. MariaDB Server (referred to as MariaDB for brevity) has similar roots as Percona Server for MySQL, but is quickly diverging from MySQL compatibility and growing on its own. MariaDB has become the default installation for several operating systems (such as Red Hat Enterprise Linux/CentOS/Fedora). Changes in the default variables can make a large difference in the out-of-box performance of the database, so knowing what is different is important.

As MariaDB grows on its own and doesn’t remain 100% compatible with MySQL, the defaults configuration settings might not mean everything or behave the way they used to. It might use different variable names, or implement the same variables in new ways. You also need to take into account that MariaDB uses it’s own Aria storage engine that has many configuration options that do not exist in MySQL.

Note: In this blog, I am looking at variables common to both MySQL or MariaDB, but have different defaults, not variables that are specific to either MySQL or MariaDB (except for the different switches inside the optimizer_switch).

Binary Logs

Variable MariaDB Default MySQL Default
sync_binlog 0 1
binlog_format Mixed Row

MySQL has taken a more conservative stance when it comes to the binary log. In the newest versions of MySQL 5.7, they have updated two variables to help ensure all committed data remains intact and identical. Binlog_format was updated to row in MySQL in order to prevent non-deterministic statements from having different results on the slave. Row-based replication also helps when performing a lot of smaller updates. MariaDB defaults to the Mixed format. Mixed uses statement-based format unless certain criteria are met. It that case, it uses the row format. You can see the detailed criteria for when the row format is used here: https://mariadb.com/kb/en/the-mariadb-library/binary-log-formats/.

The other difference that can cause a significant impact on performance is related to sync_binlog. Sync_binlog controls the number of commit groups to collect before synchronizing the binary log to disk. MySQL has changed this to 1, which means that every transaction is flushed to disk before it is committed. This guarantees that there can never be a committed transaction that is not recorded (even during a system failure). This can create a big impact to performance, as shown by a Roel Van de Paar in his blog: https://www.percona.com/blog/2016/06/03/binary-logs-make-mysql-5-7-slower-than-5-6/

MariaDB utilizes a value of 0 for sync_binlog, which allows the operating system to determine when the binlog needs to be flushed. This provides better performance, but adds the risk that if MariaDB crashes (or power is lost) that some data may be lost.

MyISAM

Variable MariaDB Default MySQL Default
myisam_recover_options BACKUP,QUICK OFF
key_buffer_size 134217728 8388608

InnoDB replaced MyISAM as the default storage engine for some time now, but it is still used for many system tables. MySQL has tuned down the MyISAM settings, since it is not heavily used.

When mysqld opens a table, it checks whether the table is marked as crashed, or was not closed properly, and runs a check on it based on the myisam_recover_options settings. MySQL disables this by default, preventing recovery. MariaDB has enabled the BACKUP and QUICK recovery options. BACKUP causes a table_name-datetime.bak file to be created whenever a data file is changed during recovery. QUICK causes mysqld to not check the rows in a table if there are no delete blocks, ensuring recovery can occur faster.

MariaDB 10.2 increased the key_buffer_size. This allows for more index blocks to be stored in memory. All threads use this buffer, so a small buffer can cause information to get moved in and out of it more quickly. MariaDB 10.2 uses a buffer 16 times the size of MySQL 5.7: 134217728 in MariaDB 10.2 vsx 8388608 in MySQL 5.7.

Innodb

Variable MariaDB Default MySQL Default
innodb_max_undo_log_size 10485760(10 MiB) 1073741824(1024 MiB)

InnoDB variables have remained primarily unchanged between MariaDB 10.2 and MySQL 5.7. MariaDB has reduced the innodb_max_undo_log_size starting in 10.2.6. This was reduced from MySQL’s default of 1073741824(1024 MiB) to 10485760(10 MiB). These sizes reflect the maximum size an undo tablespace can become before it is marked for truncation. The tablespace doesn’t get truncated unless innodb_undo_log_truncate is enabled, and it is disabled in MySQL 5.7 and MariaDB 10.2 by default.

Logging

Variable MariaDB Default MySQL Default
log_error /var/log/mysqld.log
log_slow_admin_statements ON OFF
log_slow_slave_statements ON OFF
lc_messages_dir /usr/share/mysql

Logs are extremely important for troubleshooting any issues so the different choices in logging for MySQL 5.7 and MariaDB 10.2 are very interesting.

The log_error variable allows you to control where errors get logged. MariaDB 10.2 leaves this variable blank, writing all errors to stderr. MySQL 5.7 uses an explicitly created file at: /var/log/mysqld.log.

MariaDB 10.2 has also enabled additional slow statement logging. Log_slow_admin_statements create a record for any administrative statements that are not typically written to the binlog. Log_slow_slave_statements log the replicated statements sent from the master, if they are slow to complete. MySQL 5.7 does not enable logging of these statements by default.

Lc_messages_dir is the directory that contains the error message files for various languages. The variable defaults might be a little misleading in MariaDB 10.2. Lc_messages_dir is left empty by default, although it still uses the same path as MySQL 5.7. The files are located in /usr/share/mysql by default for both databases.

Performance Schema

Variable MariaDB Default MySQL Default
performance_schema OFF ON
performance_schema_setup_actors_size 100 -1 (auto adjusted)
performance_schema_setup_objects_size 100 -1 (auto adjusted)

The performance schema is an instrumentation tool that is designed to help troubleshoot various performance concerns. MySQL 5.7 enables the performance schema, and many of its instruments, by default. MySQL even goes so far as to detect the appropriate value for many Performance Schema variables instead of setting a static default. The Performance Schema does come with some overhead, and there are many blogs regarding how much this can impact performance. I think Sveta Smirnova said it best in her blog  Performance Schema Benchmarks OLTP RW: “…test on your system! No generic benchmark can exactly repeat a workload on your site.

MariaDB has disabled the Performance Schema by default, as well as adjusted a couple of the dynamic variables. Note that if you wish to disable or enable the Performance Schema, it requires a restart of the server since these variables are not dynamic. Performance_schema_setup_actors_size and performance_schema_setup_objects_size have both been set to a static 100, instead of the dynamic -1 used in MySQL 5.7. These both limit the number of rows that can be stored in relative tables. This creates a hard limit to the size these tables can grow to, helping to reduce their data footprint.

SSL/TLS

Variable MariaDB Default MySQL Default
ssl_ca ca.pem
ssl_cert server-cert.pem
ssl_key server-key.pem

Secure Sockets Layer (SSL) and Transport Layer Security (TLS) are cryptographic protocols that allow for secure communication. SSL is actually the predecessor of TLS, although both are often referred to as SSL. MySQL 5.7 and MariaDB 10.2 support both yaSSL and OpenSSL. The default configurations for SSL/TLS differ only slightly between MySQL 5.7 and MariaDB 10.2. MySQL 5.7 sets a specific file name for ssl_ca, ssl_cert, and ssl_key. These files are created in the base directory, identified by the variable basedir. Each of these variables is left blank in MariaDB 10.2, so you need to set them before using secure connections. These variables are not dynamic, so be sure to set the values before starting your database.

Query Optimizer

MariaDB 10.2 MySQL 5.7 Optimization Meaning Switch
N/A OFF Batched Key Access Controls use of BKA join algorithm batched_key_access
N/A ON Block Nested-Loop Controls use of BNL join algorithm block_nested_loop
N/A ON Condition Filtering Controls use of condition filtering condition_fanout_filter
Deprecated ON Engine Condition Pushdown Controls engine condition pushdown engine_condition_pushdown
ON N/A Engine Condition Pushdown Controls ability to push conditions down into non-mergeable views and derived tables condition_pushdown_for_derived
ON N/A Exists Subquery Allows conversion of in statements to exists statements exists_to_in
ON N/A Exists Subquery Allows conversion of exists statements to in statements in_to_exists
N/A ON Index Extensions Controls use of index extensions use_index_extensions
OFF N/A Index Merge Allows index_merge for non-equality conditions index_merge_sort_intersection
ON N/A Join Algorithms Perform index lookups for a batch of records from the join buffer join_cache_bka
ON N/A Join Algorithms Controls use of BNLH and BKAH algorithms join_cache_hashed
ON N/A Join Algorithms Controls use of incremental algorithms join_cache_incremental
ON N/A Join Algorithms Controls use of block-based algorithms for outer joins outer_join_with_cache
ON N/A Join Algorithms Controls block-based algorithms for use with semi-join operations semijoin_with_cache
OFF N/A Join Buffer Creates the join buffer with an estimated size based on the estimated number of rows in the result optimize_join_buffer_size
ON N/A Materialized Temporary Tables Allows index creation on derived temporary tables derived_keys
ON N/A Materialized Temporary Tables Controls use of the rowid-merge strategy partial_match_rowid_merge
ON N/A Materialized Temporary Tables Controls use of the partial_match_table-scan strategy partial_match_table_scan
OFF ON Multi-Range Read Controls use of the multi-range read strategy mrr
OFF ON Multi-Range Read Controls use of cost-based MRR, if mrr=on mrr_cost_based
OFF N/A Multi-Range Read Enables key ordered scans if mrr=on mrr_sort_keys
ON N/A Order By Considers multiple equalities when ordering results ordery_uses_equalities
ON N/A Query Plan Allows the optimizer to use hidden components of InnoDB keys extended_keys
ON N/A Query Plan Controls the removal of irrelevant tables from the execution plan table_elimination
ON N/A Subquery Stores subquery results and correlation parameters for reuse subquery_cache
N/A ON Subquery Materialization Controls us of cost-based materialization ubquery_materialization_cost_based
N/A ON Subquery Materialization &

Semi-join

Controls the semi-join duplicate weedout strategy duplicateweedout

The query optimizer has several variances that not only affect query performance but also how you write SQL statements. The query optimizer is substantially different between MariaDB and MySQL, so even with identical configurations you are likely to see varying performance.

The sql_mode puts restrictions on how you can write queries. MySQL 5.7 has several additional restrictions compared to MariaDB 10.2. Only_full_group_by requires that all fields in any select…group by statement are either aggregated or inside the group by clause. The optimizer doesn’t assume anything regarding the grouping, so you must specify it explicitly.

No_zero_date, and no_zero_in_date both affect how the server interprets 0’s in dates. When no_zero_date is enabled, values of ‘0000-00-00’ are permitted but produce a warning. With strict mode enabled, then the value is not permitted and produces an error. No_zero_in_date is similar, except it applies to any section of the date(month, day, or year). With this disabled, dates with 0 parts, such as ‘2017-00-16’ are allowed as is. When enabled, the date is changed to ‘0000-00-00’ without warning. Strict mode prevents the date being inserted, unless ignore is provided as well. “INSERT IGNORE” and “UPDATE IGNORE” inserts the dates as ‘0000-00-00’. 5.7.4 changed this. No_zero_in_date was consolidated with strict mode, and the explicit option is deprecated.

The query_prealloc_size determines the size of the persistent buffer used for statement parsing and execution. If you regularly use complex queries, it can be useful to increase the size of this buffer, as it does not need to allocate additional memory during the query parsing. MySQL 5.7 has set this buffer to 8192, with a block size of 1024. MariaDB increased this value in 10.1.2 up to 24576.

Query_alloc_block_size dictates the size in bytes of any extra blocks allocated during query parsing. If memory fragmentation is a common problem, you might want to look at increasing this value. MySQL 5.7 uses 8192, while MariaDB 10.2 uses 16384 (twice that). Be careful when adjusting the block sizes: going too high consumes more than the needed amount of memory, and too low causes significant fragmentation.

The optimizer_switch variable contains many different switches that impact how the query optimizer plans and performs different queries. MariaDB 10.2 and MySQL 5.7 have many differences in their enabled options, and even the available options. You can see a brief breakdown of each of the options below. Any options with N/A is not supported in that server.

Miscellaneous

Variable MariaDB Default MySQL Default
default_tmp_storage_engine NULL InnoDB
group_concat_max_len 1048576(1M) 1024(1K)
Lock_wait_timeout 86400 (1 DAY) 31536000 (1 YEAR)
Max_allowed_packet (16777216) 16MB 4194304 (4MB)
Max_write_lock_count 4294967295 18446744073709551615
Old_passwords OFF 0
Open_files_limit 0 dependent on OS
pid_file /var/lib/mysql/ /var/run/mysqld/
secure_file_priv Varies by installation
sort_buffer_size 2097152 262144
table_definition_cache 400 autosized
table_open_cache_instances 8 16
thread_cache_size autosized autosized
thread_stack 292KB 192KB/256KB

There are many variables that do not fit well into a group. I will go over those here.

When creating temporary tables, if you do not specify a storage engine then a default is used. In MySQL 5.7 this is set to InnoDB, the same as the default_storage_engine. MariaDB 10.2 also uses InnoDB, but it is not explicitly set. MariaDB sets the default_tmp_storage_engine to NULL, which causes it to use the default_storage_engine. This is important to remember if you change your default storage engine, as it would also change the default for temporary tables. An Important note, in MariaDB this is only relevant to tables created with “CREATE TEMPORARY TABLE”. Internal in-memory temporary tables use the memory storage engine, and internal, on-disk temporary tables use the aria engine by default.

The Group_concat function can cause some very large results if left unchecked. You can restrict the maximum size of results from this function with group_concat_max_len. MySQL 5.7 limits this to 1024(1K). MariaDB increased the value in 10.2.4 up to 1048576(1M).

Lock_wait_timeout controls how long a thread waits as it attempts to acquire a metadata lock. Several statements require a metadata lock, including DDL and DML operations, Lock Tables, Flush Tables with Read Lock and Handler statements. MySQL 5.7 defaults to the maximum possible value (one year), while MariaDB 10.2 has toned this down to one day.

Max_allowed_packet sets a limit to the maximum size of a packet, or a generated/intermediate string. This value is intentionally kept small (4MB) on MySQL 5.7 in order to detect the larger, intentionally incorrect packets. MariaDB has increased this value to 16MB. If using any large BLOB fields, you need to adjust this value to the size of the largest BLOB, in multiples of 1024, or you risk running into errors transferring the results.

Max_write_lock_count controls the number of write locks that can be given before some read lock requests being processed. In extremely heavy write loads your reads can pile up while waiting for the writes to complete. Modifying the max_write_lock_count allows you to tune how many writes can occur before some reads are allowed against the table. MySQL 5.7 keeps this value at the maximum (18446744073709551615), while MariaDB 10.2 lowered this to 4294967295. One thing to note is that this is still the maximum value on MariaDB 10.2.

Old_passwords controls the hashing method used by the password function, create user and grant statements. This variable has undergone several changes in MySQL 5.7. As of 5.7.4 the valid options were MySQL 4.1 native hashing, Pre-4.1 (old) hashing, and SHA-256 hashing. Version 5.7.5 removed the “old” Pre-4.1 method, and in 5.7.6 the variable has been deprecated with the intent of removing it entirely. MariaDB 10.2 uses a simple boolean value for this variable instead of the enumerated one in MySQL 5.7, though the intent is the same. Both default the old_passwords to OFF, or 0, and allow you to enable the older method if necessary.

Open_files_limit restricts the number of file descriptors mysqld can reserve. If set to 0 (the default in MariaDB 10.2) then mysqld reserves max_connections * 5 or max_connections + table_open_cache * 2, whichever is larger. It should be noted that mysqld cannot use an amount larger than the hard limit imposed by the operating system. MySQL 5.7 is also restricted by the operating systems hard limit, but is set at runtime to the real value permitted by the system (not a calculated value).

The pid_file allows you to control where you store the process id file. This isn’t a file you typically need, but it is good to know where it is located in case some unusual errors occur. On MariaDB you can find this inside /var/lib/mysql/, while on MySQL 5.7 you will find it inside /var/run/mysqld/. You will also notice a difference in the actual name of the file. MariaDB 10.2 uses the hostname as the name of the pid, while MySQL 5.7 simply uses the process name (mysqld.pid).

Secure_file_priv is a security feature that allows you to restrict the location of files used in data import and export operations. When this variable is empty, which was the default in MySQL before 5.7.6, there is no restriction. If the value is set to NULL, import and export operations are not permitted. The only other valid value is the directory path where files can be imported from or exported to. MariaDB 10.2 defaults to empty. As of MySQL 5.7.6, the default will depend on the install_layout CMAKE option.

INSTALL_LAYOUT DEFAULT VALUE
STANDALONE,WIN NULL(>=MySQL 5.7.16_,empty(
DEB,RPM,SLES,SVR4 /var/lib/mysql-files
Other Mysql-files under the CMAKE_INSTALL_PREFIX value

Mysqld uses a sort buffer regardless of storage engine. Every session that must perform a sort allocates a buffer equal to the value of sort_buffer_size. This buffer should at minimum be large enough to contain 15 tuples. In MySQL 5.7, this defaults to 262144, while MariaDB 10.2 uses the larger value 2097152.

The table_definition_cache restricts the number of table definitions that can be cached. If you have a large number of tables, mysqld may have to read the .frm file to get this information. MySQL 5.7 auto detects the appropriate size to use, while MariaDB 10.2 defaults this value to 400. On my small test VM, MySQL 5.7 chose a value of 1400.

The table_open_cache_instances vary in implementation between MySQL and MariaDB. MySQL 5.7 creates multiple instances of the table_open_cache, each holding a portion of the tables. This helps reduce contention, as a session needs to lock only one instance of the cache for DML statements. In MySQL 5.7.7 the default was a single instance, but this was changed in MySQL 5.7.8 (increased to 16). MariaDB has a more dynamic approach to the table_open_cache. Initially there is only a single instance of the cache, and the table_open_cache_instances variable is the maximum number of instances that can be created. If contention is detected on the single cache, another instance is created and an error logged. MariaDB 10.2 suspects that the maximum eight instances it sets by default should support up to 100 CPU cores.

The thread_cache_size controls when a new thread is created. When a client disconnects the thread is stored in the cache, as long as the maximum number of threads do not exist. Although this is not typically noticeable, if your server sees hundreds of connections per second you should increase this value to so that new connections can use the cache. Thread_cache_size is an automatically detected variable in both MySQL 5.7 and MariaDB 10.2, but their methods to calculate the default vary significantly. MySQL uses a formula, with a maximum of 100: 8+ (max_connections / 100). MariaDB 10.2 uses the smaller value out of 256 or the max_connections size.

The thread_stack is the stack size for each thread. If the stack size is too small, it limits the complexity of SQL statements, the recursion depth of stored procedures and other memory-consuming actions. MySQL 5.7 defaults the stack size to 192KB on 32-bit platforms and 256KB on 64-bit systems. MariaDB 10.2 adjusted this value several times. MariaDB 10.2.0 used 290KB, 10.2.1 used 291KB and 10.2.5 used 292KB.

Conclusion

Hopefully, this helps you with the configurations options between MySQL and MariaDB. Use the comments for any questions.

PREVIOUS POST




Atomic DDL in MySQL 8.0

$
0
0

Feed: Planet MySQL
;
Author: MySQL Server Dev Team
;

The new data dictionary in MySQL 8.0 is mostly transparent to users upgrading, in that an import process is automatically run on first-start, and the semantics of DDL (creating tables, adding indexes etc) remain the same. There is however one major exception, and that is how failure cases are handled. In MySQL 8.0 DDL is now atomic.

Introduction

Some might wonder: “why has atomic DDL not been implemented before”? The matter is complicated, and parts of an answer for this is that in previous MySQL versions, persistent storage of data dictionary information have been scattered across files, non-transactional storage and transaction storage, as mentioned in MySQL 8.0 Data Dictionary: Background and Motivation .

But now that we are storing all MySQL meta data in a uniform way in  InnoDB tables,  see MySQL 8.0: Data Dictionary Architecture and Design, we are able to implement Atomic DDLs in MySQL 8.0 . This topic was presented at Percona Live 17 Dublin, in the talk: MySQL 8.0: Atomic DDLs – Implementation and Impact

Behavioral changes visible to the user

The user visible changes with the introduction of atomic DDL is tightly connected to the way crash-safeness is improved and handled in MySQL 8.0. We will discuss this later with some detailed examples below. At high level we can say that that we atomic DDL it about creating a single atomic transaction by:

  1. Do updates to the Data Dictionary (DD SE)
  2. Do necessary changes in the Storage Engine
  3. Do necessary writes to binary log

Since it is a single transaction, all updates to Data Dictionary can be rolled back and are crash safe

So for the user this will then mean that we have some changes to behavior in the following DDLs for engines supporting  atomic DDL:

  1. DROP TABLES – All tables will be dropped, or none are dropped
  2. DROP SCHEMA – All entities in the schema will be dropped, or none are dropped
  3. DROP VIEW – All views are dropped, or none are dropped
  4. CREATE USER – All users are created, or none are created
  5. DROP USER – All users are dropped, or none are dropped
  6. GRANT – All users/roles in list are granted privileges, or none

There are also other user visible changes, but they are connected to the crash-safeness handling in MySQL 8.0, that should not leave any orphaned files or index-trees. In MySQL 8.0 this is achieved by  InnoDB implementing a special internal  DDL_LOG, that during DDL execution keeps track of creation of  files and structures, which is then used at commit/rollback to clean up properly.  See below for more details on DDL_LOG.

NOTE: All of the above do require that the storage engine that is storing this information has implemented atomic DDL support. So storage engines that do not support atomic DDL, are exempted from  DDL atomicity, and partial updates to tables may still happen, as you will see in examples below. The probability for partial updates for storage engines that do NOT support atomic DDL is also significantly reduced.

The DDL_LOG in InnoDB

The DDL_LOG is a protected table in mysql tablespace to which  no DDL and no user DML is allowed. For all entries added to the  DDL_LOG a trx Id is attached, so at commit/rollback, these entried can be identified with the transaction, and proper actions can be taken. After a commit/rollback and appropriate actions by InnoDB, all entries for the transaction will be deleted from the DDL_LOG.

Changes to DDL_LOG are persisted ASAP. This prevents data files updates and a situation at a potential crash scenario where the DDL_LOG entries are not flushed. Also note that flushing of DDL_LOG is exempted from innodb_flush_log_at_trx_commit control for delayed flush.

Why is this a big deal, and how is it connected to user visible behavioral changes? Let us consider the following, related to orphaned files and index trees after a crash scenario:

Before MySQL 8.0 and pre atomic and crash-safe DDL implementation, a potential MySQL server crash could leave “orphan” InnoDB ibd files, “orphan” index trees/pages. Orphaned files/trees could then potentially cause problems for subsequent DDLs and they would also waste space in the tablespace.

With MySQL 8.0, and the new atomic and crash-safe DDL implementation, no ibd files/index trees will be orphaned if there is a crash and there is no commit of a transaction.  At commit, the old ibd files/index trees will be dropped. See examples below for more details.

Examples of implementation of Atomic and Crash Safe DDL

Let us first look at the implementation details for  CREATE TABLE as an example where there are no user visible changes:

  1. At SQL layer,  create table object, and then make an initial function call to the SE, so the SE can initiate it’s handling of the DDL.
  2. The SE adds the SE private data it has at this point in time, and returns the control to the SQL layer.
  3. Then the TABLE object is stored in the DD tables. Note that for SEs supporting atomic DDL, there is no commit at this point in time.
  4. The SQL layer can now do a an “Open table” operation, that will construct all internal structures, like TABLE_SHARE etc. The SQL layer then calls the SE handler::create(name,…).
  5. The SE creates the physical tablespaces/files(file-per-table) or Cluster index tree and other index trees. The SE then logs the above physical file and indexes creations in the DDL_LOG, and updates SE private data, and returns control to the SQL layer. Note: No separate SE transaction, all info for new tablespace/indexes are passed back to server with DD objects.
  6. The SQL layer now writes the statement to the binary log, and based on the status of the execution, it will commit or rollback the transaction.
  7. After commit/rollback, the SQL layer will call a post_ddl() hook in the SE, which enables the SE to do a proper cleanup of files and trees, and will also remove entries in the DDL_LOG for the transaction. If the transaction is rolled back, the post_ddl() physically delete the tablespace/ibd (file-per-table) and drop the index trees for the table. Note: If a crash occurs during the DDL execution, during recovery,  the SE will use the  info in DDL_LOG to delete the ibd/index trees to ensure a consistent state.

Then let us look at a more complex example DROP TABLES, which also has user visible changes, both in scenarios under normal operation, but also for crash safeness. This implementation is rather complex, as there are many parameters that influences the execution:

  1. The SQL layer for examines the list of tables in the DROP TABLES statements, and categorizes the in the following categories:
    1. Non-existent table
    2. Base table in non-atomic SE
    3. Base table in atomic SE
    4. Temporary table in a non-transactional SE
    5. Temporary table in a transactional SE
  2. If there are tables in category 1, regardless of number of files in category 2 and 3, and no IF EXISTS clause is involved, the DDL statement will err out and give a proper error message. This means there is a change in behavior even for storage engines that do not support atomic DDL.
  3. The SQL layer will then start to handle tables in category 2, if any.
    1. Call to SE handler:ha_delete_table() and the SE will the delete the necessary files, and return to SQL layer
    2. The SQL layer will drop the information about the table in the data dictionary tables and commit (this intermediate commit is to help out in a potential crash scenario)
    3. If there are no GTID assigned to the statement, a DROP TABLE statement is written to the binlog. This is done to improve crash safety for engines that do not support atomic DROP TABLE. Unlike in 5.7 this means that the difference between binary log and DD/SE contents in case of crash will be limited to at most 1 table.
  4. This is THE MAIN usecase, where the DROP TABLE statement will drop tables in an engine that supports atomic DROP TABLE, and in combination with 5 and 6 below, ensures that the statement is atomic and crash safe. The SQL layer will handle tables in category 3, if any
    1. Call to SE handler:ha_delete_table(), where InnoDB only logs actions to delete ibd file or index trees in DDL_LOG. So there is no physical action at this pint in time (No InnoDB system tables needs to be updated), return to SQL layer
    2. The SQL layer will drop the information about the table in the data dictionary tables, but will not commit at this point in time
  5. If the statement is not assigned GTID or it is assigned GTID but we have only tables from categories 1 and 3 we write DROP TABLES statement for tables from categories 1 and 3 to the binary log (category 1 can be non-empty at this point only if DROP TABLES has IF EXISTS clause).  The latter is to ensure atomicity for the case when we execute on slave in GTID mode and all tables belong to SEs supporting atomic DDL.
  6. The SQL layer then decides if the statements should be committed or rolled back, and the calls the post_ddl() hook in the SE. Note that concurrent DDL on the tables is blocked now. Note2, if a crash happens during this phase,  the post_ddl() hook actions below will be executed during recovery.
    1. In the post_ddl() hook: If commit, InnoDB physically deletes ibd and drops index trees. If rollback, delete entries in DDL_LOG, files and trees are left as they are. Note: If crash and DROP transaction is committed, the recovery process will retrieve info from the DDL_log, and delete files/trees during crash recovery.
  7. If the statement is assigned GTID and there are tables in category 2, so write to binlog is therefore postponed from item 5 above,  DROP TABLES statement with all tables we dropped is written to binlog. This is done now to handle GTID mode from pre-8.0 servers.  Note that this a compromise that is not crash safe wrt binlog/GTID update, since we cannot split a statement with GTID assigned.
  8. The SQL layer will then handle tables in category 4, if any
    1. For each temp table call SE handler::delete_table(), and the SE will delete necessary files.
    2. The SQL layer will remove the information about the temporary table from it’s internal structures
  9. Now the SQL layer will write DROP TEMPORARY TABLES for all temp tables above to the binary log
  10. The SQL layer will then handle tables in category 5, if any
    1. For each temp table call SE handler::delete_table(), and the SE will delete necessary files and trees.
    2. The SQL layer will remove the information about the temporary table from it’s internal structures
  11. Then at last the SQL layer will write DROP TEMPORARY TABLES for all temp tables above to the binary log

In the presentation MySQL 8.0: Atomic DDLs – Implementation and Impact you will find a comparison on how CREATE TABLE and DROP TABLES are handled in MySQL 5.7 . In this presentation you will also see examples on how user management atomic DDL differs from the user management DDLs in MySQL 5.7 .

Summary

As we can see from the descriptions and examples above, atomic DDL and improved crash safeness in MySQL 8.0 is a great improvement wrt to intuitive behavior and consistency. In a replicated environment this is especially important since the probability for slave drift caused by crashes during execution of DDL on master/slave is eliminated if only InnoDB is used, and  significantly reduced for SEs not supporting atomic DDL. With this new implementation in MySQL 8.0 we have eliminated possibilities for internal consistencies in the server and SE after crash.

How FINRA benefits from a cloud data lake

$
0
0

Feed: All – O’Reilly Media.
Author: John Hitchingham.

Flat Clouds Blazing Sun
Flat Clouds Blazing Sun

(source: Barta IV on Flickr)

One of the classic challenges of analytics is making data accessible to all. The data needed for analytics and data science is often locked away in different data silos, where it is difficult to discover and access. Analysts and data scientists who want to derive new insights from data within the enterprise must work with a large number of data stewards and data engineers to build their own map of data assets and source data from various data silos.

As a result of our move at FINRA to a managed data lake architecture in the cloud, our organization arrived at a solution to this problem, as well as introducing significant improvements in the flexibility of our data processing pipeline that prepares data for analytics. In this process, I’ll describe our approach.

A challenge of big data

FINRA is the Financial Industry Regulatory Authority, a not-for-profit organization authorized by Congress to protect America’s investors by making sure the broker-dealer industry operates fairly and honestly. FINRA’s Market Regulation group monitors 99% of all equity trades and 70% of all option trades in the U.S. This is done by processing billions of records a day of trade data from brokerage firms and exchanges. The data is validated, transformed, and prepared for analytic use. Once the data is ready for analytics, hundreds of automated detection models are run against the data to look for indicators of potential market manipulation, insider trading, and abuse—generating exception alerts when a pattern is matched. From there, regulatory analysts interactively delve deeper into the data to determine whether a regulatory issue exists. To stay abreast of emerging regulatory problems and develop new detection algorithms, a team of data scientists continually explores the data and develops new detection models.

To process and operate at these volumes, FINRA made early investments over a decade ago in cutting-edge, emerging data-warehouse appliances. This required a significant initial investment along with subsequent re-investments to expand capacity. Despite these investments, we still faced continual capacity challenges. Additionally, these appliances were complex to manage and operate in a dynamic business environment. Market volumes can fluctuate significantly day-to-day—sometimes by a factor of three or more. Regardless of fluctuations, FINRA must run its validation, ETL, and detection algorithms on the data within time frames specified in our service level agreements (SLAs). Investing in the capacity to meet anticipated peaks was cost prohibitive—it was not practical to dedicate constant capacity for processing the peak. As a result, when unanticipated peaks in market volume occurred, technology and business staff had to scramble to re-prioritize other business workloads in order to process the additional peak. This required additional operational support (people), monitoring 24×7 to allow intervention at the first sign of trouble.

A Hadoop alternative?

Facing these challenges, FINRA reviewed the options available at the time (2013). One emerging concept was the use of a large Hadoop cluster to create a “data lake” using the cluster’s HDFS for storage that could be a single, ever-growing store of data, combined with the processing power to query it. On the data lake, a variety of tools (at the time, Hive, Pig, and custom MapReduce) could be run against the data on this cluster. With the concept of “schema-on-read,” a schema could be applied to data at time of query, instead of time of ingest. This change allowed a variety of tools to be used for processing data—the right tool for the right job. Additionally, storage and compute capacity could be added in smaller increments using commodity hardware. This process was much better than using the data warehouse appliance, which required a major capital expenditure every few years to upgrade capacity, along with the operation complexity of moving data during upgrades and expansions.

While a Hadoop data lake appeared better than the previous state, there still remained three fundamental challenges:

  • Capacity was still not truly dynamic—While it was (relatively) easy to add capacity, this was still a process that could take days (or weeks to months if additional servers needed to be procured). Additionally, while it was possible to add capacity for peaks, it was not possible to shrink capacity once the peak had passed. We were still faced with the challenge of sizing for anticipated peak demand.
  • Capacity could not be optimized to balance storage and compute—The second challenge was that by continuing the use of a paradigm where storage and compute capacity were combined together at the node (server) level, we often had idle processing capacity that did not match our storage needs.
  • Managing a Hadoop cluster can be complex—FINRA already had experience with running Hadoop clusters in a production environment. We knew that running a multi-petabyte Hadoop cluster would be operationally complex. Operations staff experienced with Hadoop needed to continually monitor and maintain the cluster: replacing faulty hardware before it resulted in losing data, adding capacity, and managing backups. All these tasks add cost and complexity.

So, we started looking for an approach that would give us the benefits of a data lake without the operational complexity and cost.  Cloud offered the promise of pay-as-you-go resources that could be added or removed as needed based on workload. Could this address the problem of matching capacity to demand?

The cloud data lake—beyond traditional Hadoop

Eventually, we arrived at an approach of preserving the key data lake concepts: a single repository of the data, the ability to use multiple tools, “schema-on-read,” and the ability to secure the data centrally but with reduced operational complexity and cost by leveraging cloud services. This approach was accomplished by separating the key concepts of a database—catalog, query/compute engine, and storage—and projecting them onto separate cloud services:

Storage—Using an object store (e.g., AWS Simple Storage Service—S3) provides a way to store and manage data with minimal operational effort. Features that are often difficult to perform well, such as capacity management, capacity forecasting, access control, encryption, disaster recovery and archiving, are all easy to configure and automate when abstracted as a service. Adding data to our repository is as simple as storing it in S3, applying the appropriate access controls, and registering with our centralized catalog (next).

Catalog—A catalog that can store technical metadata (like DDL) is essential to a database. Additionally, it would be ideal for a catalog to also be able to contain business and operational metadata (such as lineage). We added the additional objective of a catalog that was not bound to a particular operating stack (e.g., Hadoop). Ideally, it would be a catalog that could reference data sets in other stores. After surveying the market options available at the time (early 2013), we did not find any suitable open source or commercial offering. So, we developed our own catalog, which can store technical metadata that is needed to support querying and data fixes. In addition, it features a UI that allows data scientists and other consumers to explore the data sets, layering in business metadata on top of the technical metadata. Portions of this data catalog have been open-sourced as the herd project.

Query/Compute—Having abstracted away the problem of storage management and catalog, we were left with the challenge of how to handle processing. AWS’s Elastic MapReduce (EMR) service allowed us to create and destroy Hadoop processing clusters as needed—all via simple API commands. On the cluster, we could load software (Hive originally, now Presto and Spark) that allowed our developers and analysts to use the tool they were familiar with (SQL) to access data. EMR supports the ability to run queries from these tools while keeping the data on the object storage layer (S3).  This created what was effectively a single, multi-petabyte database. SQL queries can be done from either a single compute cluster or multiple clusters running in parallel, all reading against the same data set in S3.

As I am writing this post, we have more than 70 distinct compute clusters (over 2,000 servers in total) running simultaneously in our production environment. Some clusters are tuned for user queries. Others are running cost effective batch ETL and batch analytic routines. Being able to run multiple clusters simultaneously gives us effortless ability to parallelize workloads to meet unexpected business needs while still meeting routine SLAs. Also, while we use EMR for more than 90% of our processing, our platform-independent data catalog and separate storage layer allows us to use other services for specialized processing. We use the Lambda for highly parallelized, server-less file validation. We use the Redshift database service for certain query-intensive data marts. We have flexibility in the query/compute technology we use.

Operational benefits

By re-thinking the analytic data warehouse as a series of services, we have been able to achieve significant operational benefits for our data processing. Now, when market volumes fluctuate, we can increase the capacity of our validation, ETL, and detection algorithms immediately to process the additional workload—all through a series of simple API commands. When the volume drops, we shrink the size of the processing clusters. In some cases, we can shrink them to zero if there is a lull in the incoming data stream. Because our data remains stored on S3, we leverage use of the AWS SPOT pricing market to purchase capacity at a fraction of the standard rate. If we lose a cluster due to SPOT market conditions (always a possibility), it is possible to easily re-provision the cluster and resume processing from the most recent checkpoint.

We can now easily buffer against unexpected changes in market volumes and still meet our SLAs. Additionally, it is very easy to accommodate unexpected business requests for out-of cycle processing. One condition we frequently encounter is the need to re-process data if a data error or problem is later detected in upstream data sources. These errors are outside our control and frequently require significant processing resources to accommodate. Executing them on our old data appliance infrastructure meant that they would have to wait weeks for a processing window when they could be run without interrupting processing of our normal data flows. With the managed data lake architecture, it is very easy to create additional processing clusters on EMR to execute the reprocessing at the same time the normal processing is occurring. We regularly create and destroy more than 10,000 nodes (servers) of compute capacity daily. All this allows us to complete reprocessing in a few days—not weeks.

Making access easier

While we had hoped to achieve operational benefits by moving to the new architecture, we were surprised that the approach also made our entire data set much more accessible to users such as data scientists who needed to explore the data to develop new models and insights.

Historically, it was difficult for our data scientists to explore the scope of data sets that were available and then actually obtain the data of interest for use in model development. They might need to work with multiple data engineers who were responsible for different sets of data—both to understand the data and to obtain access to the physical database where it resides. Often, the needed data had been archived to tape stored offsite, which required a time-consuming restore process. This could take weeks. Additionally, even after data sets were obtained, a data scientist might need to work with other engineers to provision a temporary data mart of sufficient capacity for them to perform their model development. This could mean weeks just to get started on model building. If there was a subsequent desire to incorporate additional data sources into the model development process, additional weeks could pass. Given the many course corrections and dead ends involved in model building, this severely limited the velocity for building new models and analytics using the data.

With the managed data lake architecture, our data scientists now have easy access to all of our data. Through the searchable interface of our data catalog, it is possible to explore all of our data sets based on business metadata to find data sets of potential interest. Also from within the catalog, it is possible to explore the technical metadata associated with a data set to learn how to query or extract it. However, the real power comes from the ability to integrate query tools with the catalog to go the next step and allow data scientist self-service access to all the data. Because all the data is stored on S3 and registered in the central catalog, a single query cluster running Presto or Spark can access all of our data. Further, it is very easy to quickly combine data sets from across the data lake and create temporary data marts for use with model development, all enabled via self-service.

Conclusion

FINRA’s entire Market Regulation application portfolio has been running in the cloud using the cloud data lake for more than a year. We have more than four petabytes (and growing) of catalog registered data accessible for query. Processing challenges that were difficult to solve are now easy, and we’ve reduced the friction our data scientists’ experience—allowing them to focus on developing new detection models to stay ahead of emerging regulatory issues.

For organizations that are having challenges keeping up with growing demand for analytics or managing exploding data volumes, rethinking the data warehouse as a virtual database based on cloud services is worth considering. If you have questions about how FINRA moved its entire Market Regulation portfolio to the cloud and how we manage and operate our environment on a daily basis, feel free to reach out to me.

Announcing Galera Cluster for MySQL 5.5.58, 5.6.38, 5.7.20 with Galera 3.22.

$
0
0

Feed: Planet MySQL
;
Author: Codership
;

Codership is pleased to announce a new release of Galera Cluster for MySQL consisting of MySQL-wsrep 5.5.58, 5.6.38, 5.7.20 and new Galera 3.22 library, wsrep API version 25.

NEW FEATURES AND NOTABLE FIXES IN THE GALERA REPLICATION LIBRARY SINCE LAST BINARY RELEASE BY CODERSHIP (3.21):

New features and notable fixes in Galera replication since last binary release

* Reporting last committed write set fixed to respect commit ordering (MW-341)

* GComm socket level error handling improved to avoid backend thread exit
in case of unexpected input from ASIO IO service (GAL-518)

* Race condition fixed in GComm message sending codepath (GAL-520)

* Fix for EVS protocol stall due to exhausted send window setting. This
bug could stall cluster messaging until the next keepalive was sent by
some node, causing intermittent pauses in write set replication. (GAL-521)

* Code changes to avoid name collisions with FreeBSD system headers (GAL-523)

Read the full release notes (how to install, repository structure) 

NOTABLE BUG FIXES IN MYSQL-WSREP:

Version MySQL-wsrep 5.7.20 and Galera 3.22, wsrep API version 25.

* Preserve –wsrep-recover log for future reference when starting the server.
The preserved log is stored in a file under MySQL data directory,
either in wsrep_recovery.ok or wsrep_recovery.fail depending on recovery
success. (MW-318)

* Avoid returning outdated values for wsrep_ready status variable (MW-384)

* A bug which caused stored procedure with an error handler to commit
a statement even in case of certification error was fixed. (MW-388)

* Crash during LOAD DATA for partition engine was fixed (MW-394)

* Fixed a crash caused by a dangling reference to wsrep status variables
array. (MW-399)

* Fixes to processing of foreign key cascades. (MW-402)

* ACL checks are now enforced before replication for all DDL operations
(MW-416)

* ALTER EVENT statement failure on slave was fixed (MW-417)

Read the full release notes  (known issues, how to install, repository structure) 

Version MySQL-wsrep 5.6.38 and Galera 3.22, wsrep API version 25

* Preserve –wsrep-recover log for future reference when starting the server.
The preserved log is stored in a file under MySQL data directory,
either in wsrep_recovery.ok or wsrep_recovery.fail depending on recovery
success. (MW-318)

* Avoid returning outdated values for wsrep_ready status variable (MW-384)

* A bug which caused stored procedure with an error handler to commit
a statement even in case of certification error was fixed. (MW-388)

* Crash during LOAD DATA for partition engine was fixed (MW-394)

* Fixed a crash caused by a dangling reference to wsrep status variables
array. (MW-399)

* Fixes to processing of foreign key cascades. (MW-402)

* ACL checks are now enforced before replication for all DDL operations
(MW-416)

Read the full release notes (known issues, how to install, repository structure)

Version MySQL-wsrep 5.5.58 and Galera 3.22, wsrep API version 25

Notable bug fixes in MySQL-wsrep:

* Avoid returning outdated values for wsrep_ready status variable (MW-384)

* Crash during LOAD DATA for partition engine was fixed (MW-394)

* Fixes to processing of foreign key cascades. (MW-402)

* ACL checks are now enforced before replication for all DDL operations
(MW-416)

Read the full release notes (known issues, how to install, repository structure)

Percona XtraDB Cluster 5.6.37-26.21-3 is Now Available

$
0
0

Feed: Planet MySQL
;
Author: MySQL Performance Blog
;

 | October 30, 2017 | 
Posted In: Events and Announcements, High-availability, MySQL, Percona Software, Percona XtraDB Cluster, ProxySQL

PREVIOUS POST

Percona XtraDB Cluster 5.6.34-26.19Percona announces the release of Percona XtraDB Cluster 5.6.37-26.21-3 on October 27, 2017. Binaries are available from the downloads section or our software repositories.

Percona XtraDB Cluster 5.6.37-26.21-3 is now the current release, based on the following:

All Percona software is open-source and free.

Fixed Bugs

  • Added access checks for DDL commands to make sure they do not get replicated if they failed without proper permissions. Previously, when a user tried to perform certain DDL actions that failed locally due to lack of privileges, the command could still be replicated to other nodes, because access checks were performed after replication.This vulnerability is identified as CVE-2017-15365.

Help us improve our software quality by reporting any bugs you encounter using our bug tracking system. As always, thanks for your continued support of Percona!

PREVIOUS POST

Alexey Zhebel

Alexey works for Percona as a Technical Writer responsible for open source software documentation. He joined the company in March 2015 with almost 8 years of prior experience in writing and managing technical documentation. Before joining Percona, Alexey worked for Oracle on Java SE and Java ME documentation. And before that his writing experience revolved around virtualization and information security software. Alexey has a degree in Electrical Engineering and a supplementary qualification in translation theory (Russian-English). He lives in Saint Petersburg, Russia with his wife and son. He spends his free time practicing and playing ultimate frisbee in a local team.

Percona XtraDB Cluster 5.7.19-29.22-3 is now available

$
0
0

Feed: Planet MySQL
;
Author: MySQL Performance Blog
;

 | October 30, 2017 | 
Posted In: Events and Announcements, High-availability, MySQL, Percona Software, Percona XtraDB Cluster, ProxySQL

PREVIOUS POST

Percona XtraDB Cluster 5.7Percona announces the release of Percona XtraDB Cluster 5.7.19-29.22-3 on October 27, 2017. Binaries are available from the downloads section or our software repositories.

NOTE: You can also run Docker containers from the images in the Docker Hub repository.

Percona XtraDB Cluster 5.7.19-29.22-3 is now the current release, based on the following:

All Percona software is open-source and free.

Fixed Bugs

  • Added access checks for DDL commands to make sure they do not get replicated if they failed without proper permissions. Previously, when a user tried to perform certain DDL actions that failed locally due to lack of privileges, the command could still be replicated to other nodes, because access checks were performed after replication.This vulnerability is identified as CVE-2017-15365.

Help us improve our software quality by reporting any bugs you encounter using our bug tracking system. As always, thanks for your continued support of Percona!

PREVIOUS POST

Alexey Zhebel

Alexey works for Percona as a Technical Writer responsible for open source software documentation. He joined the company in March 2015 with almost 8 years of prior experience in writing and managing technical documentation. Before joining Percona, Alexey worked for Oracle on Java SE and Java ME documentation. And before that his writing experience revolved around virtualization and information security software. Alexey has a degree in Electrical Engineering and a supplementary qualification in translation theory (Russian-English). He lives in Saint Petersburg, Russia with his wife and son. He spends his free time practicing and playing ultimate frisbee in a local team.

Bruce Momjian: Blocking DDL

$
0
0

Feed: Planet PostgreSQL.

This blog is about my work on the Postgres open source
database, and is published on Planet PostgreSQL.
PgLife allows monitoring of
all Postgres community activity.

Conference Why Attend Conferences?

Friday, November 3, 2017

I have attended 323 Postgres events in my career. While I have enjoyed almost all of
them, many had different focuses, so I thought I would share my experiences. First, there are a variety of conference types:

  1. Vendor conferences: often in big cities, which focus on company-produced products
  2. Business conferences: also often in big cities, which focus on business challenges and discussions, often with high attendance prices
  3. Community conferences: led by people who care about open-source software and focus on software knowledge transfer
  4. Hobbyist conferences: often in smaller cities, which focus on interpersonal relationship building with technology as a catalyst, often free

It would be nice if I could say which conference types are good or bad, but that isn’t possible. Each conference targets an audience whose needs it seeks to fulfill. Let’s look at
the needs that each fulfills:

  1. Vendor conferences: If you are new to a technology and need people to help you navigate purchase options, these conferences are for
    you.
  2. Business conferences: If you are frequently challenged to make business decisions, but feel you have no one to share options with or
    brainstorm, this type of conference can give you a framework to help you make your next complex business decision.
  3. Community conferences: If you spend significant time solving technological problems, you can gain great insight and new approaches by
    attending this type of conference.
  4. Hobbyist conferences: If you are looking for emotional connections to people who share similar interests, this type of conference can
    be personally rewarding.

Ideally everyone would go to conferences which match their interests, but what happens when they don’t match? Here are some examples:

  1. Vendor conferences: “Wow, this is boring. The booth staff don’t even know about the technology they are selling. When will this be
    over?”
  2. Business conferences: “People are very passionate about the problems they are trying to solve. I am glad I don’t have these problems
    — they seem unsolvable.”
  3. Community conferences: “These people really care about the minutia of the software. When are they going to get a life?”
  4. Hobbyist conferences: “Does this end with everyone sitting in a circle and roasting marshmallows over a cpu fan?”

Ninety-five percent of Postgres conferences are community conferences (#3). They have some of the other aspects, but that is not the focus. Open source vendor conferences,
e.g. LinuxWorld, used to be popular but are almost extinct. They were really focused on vendors, and
when attendees realized this, and vendors didn’t get the desired sales opportunities, demand collapsed. Postgres has few business-focused conferences —
Postgres Vision is an example. Hobbyist conferences still exist, and probably always will, though they
are, by necessity, small.

For me, conferences allow me to hear from and talk to people knowledgeable in a field I want to study. These interactions convey
information I can’t get from reading books and articles. The best interactions are a fire hose of information that I can absorb and
probe. Such interactions give me information it would take me weeks or months to learn.

This Slashdot discussion covers some of the value in attending
conferences. Hopefully this blog post and that discussion will help you navigate the many conferences offered and help you get the best value from them.


Towards Bigger Small Data

$
0
0

Feed: Planet MySQL
;
Author: Yoshinori Matsunobu
;


 In MySQL ecosystems, it has been a very common practice to shard MySQL databases by application keys, and to manage multiple small sized MySQL instances. Main reason of the sharding was that a single server could not handle so much data. 10 years ago, typical commodity servers had only 16GB RAM, and typical storage configuration was RAID10 with 6~12 SATA HDDs. There was no affordable flash storage available at that time. Because such machines could handle only a few hundred random IOPS, and buffer pool was so limited, we couldn’t put much data on a single server — at most a few hundred GBs per server. Even small-mid sized applications easily exceeded single server capacity. They had to split databases into multiple servers.

Disadvantages of Sharding

 Sharding by applications has been a common practice to scale MySQL beyond single machine. But there are a couple of disadvantages like the followings.

  • You have to write application logic to manage shards. Also, you need to manage many more MySQL instances
  • Atomic transaction is not supported across multiple shards
  • Can not take globally consistent backups across multiple shards
  • Cross instance join is not supported (or very slow)
  • Hard to use secondary keys efficiently. Unless secondary keys are part of sharding keys, you need to query all shards, which is very expensive, especially if you need sorting

 Atomic transaction, join, and secondary keys are big deals. Not all applications can easily give up them. They might not be willing to spend time to implement sharding logic, either.

Shared Nothing SQL database is not general purpose

 There are a couple of database products that offer transparent sharding by database internals. MySQL Cluster (NDB) is one of the MySQL engines that has existed for long years, offering shared nothing distributed databases. You can run atomic transactions, run cross-shard joins across multiple instances (data nodes). MySQL Cluster supports NoSQL interface (it is called NDBAPI), to query directly into data nodes, which boost performance significantly. I’m not going to talk about the NoSQL interface, since I’d like to discuss SQL database. Note that MySQL Cluster is a product name and it is not InnoDB. If you want to continue to use InnoDB, Vitess can be a good solution. It helps to build transparent, shared nothing database.
 Transparent shared nothing databases like MySQL Cluster, Vitess solved some of the issues mentioned above, but there are still issues like below, and users may need to redesign tables and/or rewrite queries as needed.

  • Limited secondary key support. This was the same as I mentioned above. In MySQL Cluster, by default, rows are distributed by primary keys. If you do not explicitly specify primary keys in WHERE clause, queries need to scan all data nodes, which may significantly limit concurrency and increase latency, depending on how many data nodes you have and other query conditions (e.g. sorting).
  • Queries using JOINs may need lots of network tound-trips. Lots of improvements have been done in MySQL Cluster for handling joins, but it’s still slower than InnoDB, especially if queries are complex.

 As a result, to utilize MySQL Cluster, it was generally recommended not using secondary keys, and not using joins frequently. There are many good use cases, especially if people are 100% sure how their databases are used, but it is not recommended as a general purpose SQL database.

Bigger Small Data and its challenges

 Several people at Facebook called MySQL services as “Small Data”. Combined MySQL instance size was pretty large, but each instance size was small enough (normally up to 1TB). 10 years ago, people had to run small instances because of limited hardware support. Nowadays, commodity servers have more than 256GB RAM and more than 10TB Flash storage. There are many small-mid databases that fit in 10TB. Successful databases grow beyond 10TB, so they will have to shard anyway. But how about experimental projects, and/or many other applications that are expected to grow up to limited size? Instead of spending engineering efforts to manage shards and rewrite tables/queries, why not just put everything on a single server and take all advantages like atomic and consistent transactions, secondary indexes and joins — running “Bigger Small Data”?

 There are a couple of public MySQL services supporting bigger storage size. Amazon Aurora (AWS) and Alibaba PolarDB (Alibaba Cloud) are both based on MySQL and claim to support more than 60TB instance size. It was not surprising to me that they chose bigger small data rather than shared nothing distributed database, because they had lots of customers who wanted to do whatever they wanted. They couldn’t control customers not to use joins.
But being supported does not necessarily mean working as expected. To make MySQL really work with bigger small data, it needs lots of improvements, beyond 8.0, especially improving concurrency and long running operations, including the followings.

  • Parallel query
  • Parallel binary copy
  • Parallel DDL
  • Resumable DDL
  • Better join algorithm
  • Much faster replication
  • Handling many more connections
  • Good resource control, so that some bad users don’t eat all resources

 These are needed at least, to answer questions like “how can MySQL handle general operations, if our instance grows 10x bigger”?

 I’m not worried about short queries — row look-ups by primary keys or secondary keys. These are where MySQL has been great so far. I’m worried about long running queries. The most common case of the long running queries would be full table scan from a single table. In general, logical dump from 10TB table takes (much) more than 10 times, compared to scanning from 1TB table. InnoDB needs to keep history list for consistent reads. Maintaining history list and reading consistent snapshots from rollback segments get more expensive, if rows are heavily updated during running long running transactions. If you run daily backups by mysqldump, you might not tolerate 10x~20x longer backup time — which might not finish in 24 hours. To make logical dump shorter, parallel query support is needed, and this is not currently not supported by MySQL. Physical backup (binary copy) also needs parallelism, though it can be relatively easily implemented, since physical backup tools are written out side of MySQL and are easily extended.

 Running ALTER TABLE on 10TB table is also challenging. Amazon Aurora supports instant DDL — adding a nullable column at the end of a table can be done without rebuilding the table. But there are still many DDL operations requiring copying tables. First, you will want to know when it ends. MySQL currently does not tell that. If it is expected to take 20 days, you might be worried what will happen if mysqld restarts before finishing the DDL. It would be great if the database remembers DDL state periodically, and can resume operations when restarting mysqld.

 Replication is another technical challenge. MySQL replication is asynchronous. Slaves are often lagged if master instances are heavily updated. On bigger small data, update volume on master can be 10x or even more. How slaves can handle 10x more replication traffics?
Amazon Aurora does not have MySQL replication lag issue, if you run it in a single region. Aurora has 6x storage copies in the same region, across three availability zones. You can scale reads in the same region. But scaling reads across multiple regions requires MySQL Replication, and it is challenging unless making MySQL Replication a lot faster. Alibaba PolarDB offers InnoDB physical replication across different datacenters, which is significantly more concurrent and faster than binlog based replication. But you are constrained to one engine (InnoDB, though it’s by far the most common engine) and debugging replication issues might be harder, since it’s no longer binlog based.

 Reliability improvement should not be ignored. 10x larger data means the instance serves many more connections and queries. If handful bad queries take the whole instance unavailable, the impact in bigger small data is much higher than in small instances. Good resource management is needed. High priority queries can be useful too, for making some important low latency queries finish earlier, without being affected by expensive queries.

 There are lots of technical challenges to make Bigger Small Data really work. I expect Amazon Aurora will be ahead to implement these important features to make Aurora truly support bigger data. But I’m almost sure that AWS won’t release them as open source software. For everybody to get such features, somebody else will have to implement. I hope Oracle will do, but I understand that they need to compete Amazon Aurora and Oracle will not be willing to give their features to AWS for free. More realistic scenario might be multiple companies, including us, implementing features, releasing as open source and contributing to Oracle and/or MariaDB. I think these are interesting technical projects for MySQL for a couple of years.

Oracle Upgrades to 12.2.0.1

$
0
0

Feed: Databasejournal.com – Feature Database Articles.
Author: .

With the recent release of Oracle version 12.2.0.1 some shops will be planning and executing upgrades; part of this may be the availability of expanded JSON functionality, APEX improvements and online table management such as moves and partition splits. Careful planning is crucial to a successful upgrade, including object management tasks that may not seem important but can send an upgrade ‘sideways’. Let’s look at those possibly overlooked items.

Depending upon which release the upgrade starts from the path may become longer than expected, traveling through more than one Oracle release before finally ending up at 12.2.0.1. Upgrades from versions lower than 11.2.0.3 will necessitate getting to release 11.2.0.3 or 11.2.0.4 before proceeding further; it shall be presumed that the database is at either of those two releases for this discussion.

The obvious place to begin is with the pfile or spfile; deprecated parameters should be removed or commented to prevent them from being set. The following list shows all but one of the parameters deprecated by Oracle for releases up to and including 12.2.0.1:


active_instance_count
background_dump_dest
buffer_pool_keep
buffer_pool_recycle
commit_write
cursor_space_for_time
fast_start_io_target
global_context_pool_size
instance_groups
lock_name_space
log_archive_start
max_enabled_roles
parallel_automatic_tuning
parallel_io_cap_enabled
parallel_server
parallel_server_instances
plsql_debug
plsql_v2_compatibility
rdbms_server_dn
remote_os_authent
resource_manager_cpu_allocation
sec_case_sensitive_logon
serial_reuse
sql_trace
standby_archive_dest
user_dump_dest

The missing parameter is sec_case_sensitive_logon and is unsupported in what Oracle calls ‘Exclusive Mode’, a reference to how the SQLNET.ALLOWED_LOGON_VERSION_SERVER parameter is set (if it’s set at all) in the sqlnet.ora file. By default, it’s set to 12 so that only version 12 clients can connect to the database. It can be set to values as low as 8 (to ensure backward compatibility for older applications) and it also affects accounts with a password version less than 11G. For compatibility reasons case-insensitive passwords are required for some older applications or applications that access both a 12.x database and an older release. The pre-upgrade tool provided by Oracle will report such issues, so they can be addressed prior to the upgrade. The command to run the tools is:


$ORACLE_HOME/jdk/bin/java -jar {Oracle 12.2 home}/rdbms/admin/preupgrade.jar FILE TEXT DIR {directory to write the output to}

The $ORACLE_HOME variable, and the corresponding environment, must be set to the current, pre-12.2 Oracle home for the database to be upgraded. The tool is run from the 12.2 home. That utility will generate the following scripts and directories:


dbms_registry_basic.sql         postupgrade_fixups.sql          preupgrade.log                  upgrade
dbms_registry_extended.sql      preupgrade_driver.sql           preupgrade_messages.properties
oracle                          preupgrade_fixups.sql           preupgrade_package.sql

The preupgrade.log has all of the information necessary to prepare the database for upgrade. The .sql scripts generated tell you what they are for and the log will report which scripts to run to prepare the database. [For those who are curious, setting SQLNET.ALLOWED_LOGON_VERSION_SERVER to 8 ‘turns off’ Exclusive Mode so accounts with a password version of 10G will still be accessible; not setting SQLNET.ALLOWED_LOGON_VERSION_SERVER when accounts have a 10G version password (such as those accounts created by a legacy application) will render them no longer accessible.] There may be post-upgrade actions to take (such as upgrading the database timezone) which will be reported once the postupgrade_fixups.sql script has run.

Invalid objects can interfere with an upgrade in such a way as to cause ORA-00600 errors to be generated. To be honest it is a special case that can cause this: views, materialized views, packages, functions and procedures that use database links that no longer exist can send an upgrade into a downward spiral. Obviously since these are already invalid and cannot be successfully compiled it’s highly likely they are not being used; another case however, is when a test, dev or UA database is created from a production database and such database links are dropped for security reasons. It will still be necessary to find these objects (utlrp.sql will report them in the DBA_ERRORS table) so the DDL can be preserved and the objects dropped prior to starting the upgrade. After the upgrade is successfully completed these objects can be recreated, although they will still be invalid.

The dba_recyclebin MUST be purged prior to starting an upgrade, as the upgrade process accesses the data dictionary for the database undergoing the upgrade and recyclebin objects can stop an upgrade or cause the database to throw unexpected errors after the upgrade has apparently completed successfully.

Unused columns are also an issue as the data dictionary keeps track of them and sets their position in the table to 0; the upgrade process doesn’t check for a position value of 0, it simply checks the dictionary for columns that may need to be upgraded because of features that have been selected. Unlike invisible columns that CAN be used after they are set to INVISIBLE, dropped columns have no useful names and cannot be recovered. If developers have set columns to UNUSED then they are really, truly gone and need to be dropped. A query of DBA_UNUSED_COL_TABS will provide a list of the owners, tables and number of columns marked UNUSED. It’s a simple task to use that view to create a script to rid those tables of their unused baggage:



select 'alter table '||owner||'.'||table_name||' drop unused columns;'
from dba_unused_col_tabs

spool drop_unused.sql
/
spool off

spool drop_unused_cols.log
set echo on
@drop_unused
set echo off
spool off

So the upgrade won’t fail.

Backup the database BEFORE starting an upgrade; should the upgrade fail, the database can be restored into its original home so that the business isn’t without it during the ‘investigate the failure’ phase. Do not ignore this step; it has saved many a DBA and enterprise from losing a database to a failed upgrade.

It is no longer possible to run the catupgrd.sql script directly from the SQL> prompt; it’s a Java script that is called as shown below:


$ORACLE_HOME/perl/bin/perl $ORACLE_HOME/rdbms/admin/catctl.pl $ORACLE_HOME/rdbms/admin/catupgrd.sql

which runs in a mixture of serial and parallel modes; each step, and the mode used, are reported at the terminal screen during the upgrade. Sample output is shown below:


------------------------------------------------------
Phases [0-115]         Start Time:[2017_09_21 13:59:45]
------------------------------------------------------
***********   Executing Change Scripts   ***********
Serial   Phase #:0    [{sid}] Files:1    Time: 146s
***************   Catalog Core SQL   ***************
Serial   Phase #:1    [{sid}] Files:5    Time: 54s
Restart  Phase #:2    [{sid}] Files:1    Time: 0s
***********   Catalog Tables and Views   ***********
Parallel Phase #:3    [{sid}] Files:19   Time: 18s
Restart  Phase #:4    [{sid}] Files:1    Time: 0s
*************   Catalog Final Scripts   ************
Serial   Phase #:5    [{sid}] Files:6    Time: 22s
*****************   Catproc Start   ****************
Serial   Phase #:6    [{sid}] Files:1    Time: 22s
*****************   Catproc Types   ****************
Serial   Phase #:7    [{sid}] Files:2    Time: 21s
Restart  Phase #:8    [{sid}] Files:1    Time: 1s
****************   Catproc Tables   ****************
Parallel Phase #:9    [{sid}] Files:69   Time: 35s
Restart  Phase #:10   [{sid}] Files:1    Time: 1s
...

When the upgrade completes, the total time is reported and the names and locations of the log file and summary report are displayed. These files need to be checked before moving on to any post-upgrade activities as any errors encountered or generated during the upgrade will be reported in these files; no such output is displayed on the terminal during the upgrade process. If the steps recommended by the preupgrade tool have been followed it’s very unlikely that the upgrade will fail; obviously there is the odd chance that something in the database configuration specific to an application can cause problems but the ‘odds’ ase in the DBA’s favor if the preupgrade instructions are followed.

Upgrading to version 12.2.0.1 is different from previous upgrades but it doesn’t have to end in tragedy. Using the preupgrade tool (as Oracle recommends), following the instructions provided by that tool and addressing the issues listed here will go a long way toward making a 12.2.0.1 upgrade a success.

See all articles by David Fitzjarrell

GoDaddy: Customer-First Digital Transformation

$
0
0

Feed: Alation.
Author: Ron Powell.

This article is based on a podcast Ron Powell conducted with Sharon Graves, Enterprise Data and BI Tools Evangelist for GoDaddy, about data curation, data stewardship, and data catalogs.  Ron is an independent analyst and industry expert for the BeyeNetwork and executive producer of The World Transformed Fast Forward Series. His focus is on business intelligence, analytics, big data, and data warehousing.

GoDaddy is committed to being a customer-first
organization. Can you explain what that means for your customers and your
employees?

Sharon Graves: Certainly. What I’ve seen as GoDaddy has
evolved over the nine years that I’ve been there is that we used to just put
out products. We did not always follow through to see if our customers were
activating the tools and proceeding along with the process to ensure that they
were actually getting to a final success state. Now, we’re actually capturing
the appropriate data to make sure that our customers are activating, setting
up, building their website, or adding a shopping cart – that they’re proceeding
down the path so they can be successful in building their web presence. That,
in turn, makes us successful.

You’re really involved in the engagement, and
the more you’re involved in the engagement and helping them, the more the product will stick.

Graves: Absolutely. If they’re successful, we’ll be successful.

What role does data play in your customer-first culture?

Graves: As I mentioned, one of the key things for us is that we sell web products for our customers to build their own web presence – domains, hosting, shopping carts, and SSL certs.  We want to make sure that we’re capturing the data points to make sure that, for example, setup is easy. Are they stopping somewhere in setup? Maybe we have a step in the process that’s too cumbersome or too convoluted. If not, maybe they just aren’t taking the next step. Maybe they need to be encouraged, or maybe they need help. Or maybe they’ve set up a shopping cart, but they never set up an SSL for it. We can monitor all of that, look at all the data around our customers, and say to them, “You’ve set up a shopping cart. Let’s set you up with security now and make sure that you’re building an appropriate site.”

Capturing
all of the data about everything that goes on in their customer world really
helps us drive our customer success.

I understand that your transformation at
GoDaddy involved getting the right customer data consolidated in one place and
making it accessible for every employee for data-driven decision making. What
did that involve?

Graves: One of the things that we found with our user population within GoDaddy, our business partners, is that we needed to get them information that they could easily use via self-service. We have a BI/BA organization that we wanted to keep focused on the very deep dive, heavy analytics and data science, but we needed to give our business partners the tools so they could do their own reporting, see the successes of their products and see the steps they needed to take. So we came up with a concept called a Unified Data Set (UDS). These data sources are packaged, managed and curated. They are groupings of data – order data, visit data or chat data etc. The intention is that our business partners should be able to answer 80 percent of their questions about their customers within this data source. That’s how we continue to grow that. It frees up our analytic side for the deeper dive but also gives our partners the deep data that they need.

Sharon, how were you able to get all your information
in one place?

Graves: One of the tools that we’re utilizing is Alation, which is a data catalog tool. What it does for us is that it goes out and pulls all of the metadata in from a variety of platforms that we have data stored on, and it actually allows us to collaborate and define and propagate out information about what the data is. One of the issues that we had is that we built great data sources and enabled end users with a reporting tool, but we didn’t really tell them where it was. We needed to actually expand on that. So with Alation, our users can search for a data source or even search for a topic, and it will point them in the right direction to use for any analysis that they might want to perform.

From the standpoint of a data catalog and
Alation, what are the benefits that it provides for you?

Graves: One of the biggest challenges with data that I’ve seen in my IT experience is that developers will create tables and processes, but nobody wants to do the documentation. Nobody wants to define what your data is because it’s cumbersome. It’s difficult. So having a data catalog such as Alation that can go out and actually pull in the foundation data, the schemas, the tables, the columns and the rows gives you a foundation that you can easily collaborate on top of to build out your documentation. It gives you a shell that’s much better than anything that you’ve had in the past where somebody just said, “I built this table. Here’s the DDL for it. Here’s just a list of the fields.”

Alation actually allows us to go out
and define it. We can actually tell the end users how they might want to use
it. We can tell them how other people are using this data source. We can
actually point them to people if they have a question. They can look at filters
that are being applied, and we can tell them which are the right filters to
apply and which are not. We can tell them how they’re joining in this data
source, and we can tell them whether that’s right or not. By having a
collaborative environment where people are being data stewards on this data and
identifying the appropriate way to use the data and what the data is for really
helps expand the knowledge and usage of the data within our ecosystem.

There’s an old saying that by getting started,
you’re half way done. So in reality, just getting the project started, Alation
actually does 50% of the work.

Graves: Absolutely. It gives you the shell so people then aren’t afraid to continue on.

What are the main benefits GoDaddy is
receiving as a result of your self-service journey?

Graves: First of all, our products are performing much better than they had in the past. We are more aware of any stumbling blocks that we may be causing for our external customers and more aware of what they may want. We can see their usage patterns and identify that there is something we should look into because a lot of people are asking about it. We’re really becoming better at understanding what our customers need and want.

Internally, we have the additional
benefit in that we have our BI/BA organization analysts really doing true data
science and looking into the hard questions as opposed to answering questions
such as how did my product trend last year and what was my year over year
variance. We now have our end users doing that themselves.

In addition, now they’re thinking more
about data whenever they’re starting to make a change or add a product. They’re
thinking about how they’ll be able to track it once it’s in place. From what
I’ve seen in the IT world in the past, a lot of people would implement products
or changes without thinking about how to track the success of it until it was
in. Then they would have to back-build. So this is a huge hurdle our internal customers
have jumped! Our internal customers are actually thinking about these things.
They’re thinking immediately how they’re going to report it, how they’re going
to show it in Tableau, and where the data is going to come from to prove out
that it’s working successfully. This has changed the mind-set within our
company.

From a data source perspective, how many data
sources do you have?

Graves: As far as the curated ones, we’re running about 30. Again, they’re very topic focused – traffic, orders, chat and things like that. We have thousands, obviously, if you talk about the entire ecosystem of GoDaddy. Not all of them are curated. That’s where we’re trying to work on our balance of how far do we document, how far do we go, and should we present all of these or not. We’re working through that governance aspect right now.

Can you explain data curation and data
stewardship at GoDaddy and what it means for the company, employees and
customers?

Graves: For me, data curation was the result of the UDSs. We have data sources, again, that we feel will answer 80 percent of the questions. We work with the BI/BA organization to determine what should be in this data source to answer that, to address this, and to alleviate that from you. What can we provide your end users so they can answer these questions?

Secondly,
now that we’ve built these data sources with our engineering team, we want to
make sure that it’s going through the proper tools. If it’s going through
change control and change is recommended, we want to make sure somebody is
doing peer review on the code, that it’s documented and that it is scheduled.
In addition to that, we have the monitoring and alerting, so we’re tracking to
see if there are variances in the data that don’t look normal – if there are
trending anomalies. Is the data current as of the expected time? Is it through
last hour or is it through yesterday as it should be, depending on the data? We
monitor that and make sure that the data is available and present for our end
users.

In
addition to that, in the data catalog we have the documentation spelling out
what the table is for, spelling out what the fields are, how they’re calculated
if they are calculated, building out the data dictionary to tie back to the
data, to tie back to the table, to tie back to the fields – saying this is how
it all is and this is what this all means. To me, that would be a curated data
source. And, again, that builds confidence in our end users that they’re using
the right data, they’re understanding how they’re using that data, and they
know that they’re on the right path.

How do you ensure the data is right?

Graves: Again, we’re monitoring. Right now if certain KPIs are out of variance, we’ll alert our on-call team and they’ll check to determine the reason for the variance. We also track if a data source is current. For example, if it is supposed to be current as of the last three hours but is not, we alert somebody to check the job to be sure it’s running. So currently we monitor source to target variances, making sure everything is tying as it should be.

As you’ve been going through this, has
the number of users using the data increased?

Graves: It is increasing substantially. We actually have some trending where we monitor the usage of our UDS data sources, and it’s in a constant climb.

That sounds wonderful. So you’re
making everyone an analyst!

Graves: Yes, and that, in turn, helps our end customer because everyone is thinking about improving and how we can change and grow the company and support our customers even better.

From a growth perspective, how is
GoDaddy doing?

Graves: We’re doing very well and our end customers are successful.

From a culture perspective, have you
seen a change?

Graves: Early on, our culture was very tech-focused and we have evolved into a customer-focused company. So that has become our real focus – making sure that our customers are happy with our tools, are happy with our products and are happy with our support. That’s one of the key metrics that we’re always measuring. If they’re happy, we’re going to be good.

So from a technical perspective, a
business user doesn’t really have to be very technical?

Graves: No, and that’s what’s been great. For Tableau, you don’t have to be very technical. It’s drag and drop. It’s very easy to use. We’re providing our customers with curated data sources that are consistent in terminology so they recognize the fields. Again, they know what they’re using. Then we have Alation that can define it further for them if they need to. It’s really all very easy to understand and easy to use.

It’s great to see a customer-first
culture. Thank you Sharon.

InnoDB Online DDL Operations with ALGORITHM=INPLACE

$
0
0

Feed: MariaDB Knowledge Base Article Feed.
Author: .

Column Operations

ALTER TABLE ... ADD COLUMN

InnoDB supports adding columns to a table with ALGORITHM set to INPLACE.

The table is rebuilt, which means that all of the data is reorganized substantially, and the indexes are rebuilt. As a result, the operation is quite expensive.

With the exception of adding an auto-increment column, this operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab ADD COLUMN c varchar(50);
Query OK, 0 rows affected (0.006 sec)

This applies to ALTER TABLE ... ADD COLUMN for InnoDB tables.

ALTER TABLE ... DROP COLUMN

InnoDB supports dropping columns from a table with ALGORITHM set to INPLACE.

The table is rebuilt, which means that all of the data is reorganized substantially, and the indexes are rebuilt. As a result, the operation is quite expensive.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab DROP COLUMN c;
Query OK, 0 rows affected (0.021 sec)

This applies to ALTER TABLE ... DROP COLUMN for InnoDB tables.

ALTER TABLE ... MODIFY COLUMN

This applies to ALTER TABLE ... MODIFY COLUMN for InnoDB tables.

Reordering Columns

InnoDB supports reordering columns within a table with ALGORITHM set to INPLACE.

The table is rebuilt, which means that all of the data is reorganized substantially, and the indexes are rebuilt. As a result, the operation is quite expensive.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab MODIFY COLUMN c varchar(50) AFTER a;
Query OK, 0 rows affected (0.022 sec)

Changing the Data Type of a Column

InnoDB does not support modifying a column’s data type with ALGORITHM set to INPLACE in most cases.

In MariaDB 10.2.2 and later, the only exception is that increasing the size of a VARCHAR column, which is performed as an instantaneous metadata update.

For example, this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab MODIFY COLUMN c int;
ERROR 1846 (0A000): ALGORITHM=INPLACE is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY

But this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab MODIFY COLUMN c varchar(100);
Query OK, 0 rows affected (0.005 sec)

Changing a Column to NULL

InnoDB supports modifying a column to allow NULL values with ALGORITHM set to INPLACE.

The table is rebuilt, which means that all of the data is reorganized substantially, and the indexes are rebuilt. As a result, the operation is quite expensive.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50) NOT NULL
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab MODIFY COLUMN c varchar(50) NULL;
Query OK, 0 rows affected (0.021 sec)

Changing a Column to NOT NULL

InnoDB supports modifying a column to not allow NULL values with ALGORITHM set to INPLACE. It is required for strict mode to be enabled in SQL_MODE. The operation will fail if the column contains any NULL values. Changes that would interfere with referential integrity are also not permitted.

The table is rebuilt, which means that all of the data is reorganized substantially, and the indexes are rebuilt. As a result, the operation is quite expensive.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab MODIFY COLUMN c varchar(50) NOT NULL;
Query OK, 0 rows affected (0.021 sec)

Adding a New ENUM Option

InnoDB supports adding a new ENUM option to a column with ALGORITHM set to INPLACE. In order to add a new ENUM option with ALGORITHM set to INPLACE, the following requirements must be met:

  • It must be added to the end of the list.
  • The storage requirements must not change.

This operation only changes the table’s metadata, so the table does not have to be rebuilt..

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c ENUM('red', 'green')
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab MODIFY COLUMN c ENUM('red', 'green', 'blue');
Query OK, 0 rows affected (0.004 sec)

But this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c ENUM('red', 'green')
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab MODIFY COLUMN c ENUM('red', 'blue', 'green');
ERROR 1846 (0A000): ALGORITHM=INPLACE is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY

Adding a New SET Option

InnoDB supports adding a new SET option to a column with ALGORITHM set to INPLACE. In order to add a new SET option with ALGORITHM set to INPLACE, the following requirements must be met:

  • It must be added to the end of the list.
  • The storage requirements must not change.

This operation only changes the table’s metadata, so the table does not have to be rebuilt..

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c SET('red', 'green')
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab MODIFY COLUMN c SET('red', 'green', 'blue');
Query OK, 0 rows affected (0.004 sec)

But this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c SET('red', 'green')
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab MODIFY COLUMN c SET('red', 'blue', 'green');
ERROR 1846 (0A000): ALGORITHM=INPLACE is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY

Removing System Versioning from a Column

In MariaDB 10.3.8 and later, InnoDB supports removing system versioning from a column with ALGORITHM set to INPLACE. In order for this to work, the system_versioning_alter_history system variable must be set to KEEP. See MDEV-16330 for more information.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50) WITH SYSTEM VERSIONING
);

SET SESSION system_versioning_alter_history='KEEP';
SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab MODIFY COLUMN c varchar(50) WITHOUT SYSTEM VERSIONING;
Query OK, 0 rows affected (0.005 sec)

ALTER TABLE ... ALTER COLUMN

This applies to ALTER TABLE ... ALTER COLUMN for InnoDB tables.

Setting a Column’s Default Value

InnoDB supports modifying a column’s DEFAULT value with ALGORITHM set to INPLACE.

This operation only changes the table’s metadata, so the table does not have to be rebuilt.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.
For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab ALTER COLUMN c SET DEFAULT 'No value explicitly provided.';
Query OK, 0 rows affected (0.005 sec)

Removing a Column’s Default Value

InnoDB supports removing a column’s DEFAULT value with ALGORITHM set to INPLACE.

This operation only changes the table’s metadata, so the table does not have to be rebuilt.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50) DEFAULT 'No value explicitly provided.'
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab ALTER COLUMN c DROP DEFAULT;
Query OK, 0 rows affected (0.005 sec)

ALTER TABLE ... CHANGE COLUMN

InnoDB supports renaming a column with ALGORITHM set to INPLACE, unless the column’s data type or attributes changed in addition to the name.

This operation only changes the table’s metadata, so the table does not have to be rebuilt.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab CHANGE COLUMN c str varchar(50);
Query OK, 0 rows affected (0.006 sec)

But this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab CHANGE COLUMN c num int;
ERROR 1846 (0A000): ALGORITHM=INPLACE is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY

This applies to ALTER TABLE ... CHANGE COLUMN for InnoDB tables.

Generated Columns

Generated columns do not currently support online DDL for all of the same operations that are supported for “real” columns.

See Generated (Virtual and Persistent/Stored) Columns: Statement Support for more information on the limitations.

Index Operations

ALTER TABLE ... ADD PRIMARY KEY

InnoDB supports adding a primary key to a table with ALGORITHM set to INPLACE.

If the new primary key column is not defined as NOT NULL, then it is highly recommended for strict mode to be enabled in SQL_MODE. Otherwise, NULL values will be silently converted to the default value for the given data type, which is probably not the desired behavior in this scenario.

The table is rebuilt, which means that all of the data is reorganized substantially, and the indexes are rebuilt. As a result, the operation is quite expensive.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int,
   b varchar(50),
   c varchar(50)
);

SET SESSION sql_mode='STRICT_TRANS_TABLES';
SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab ADD PRIMARY KEY (a);
Query OK, 0 rows affected (0.021 sec)

But this fails:

CREATE OR REPLACE TABLE tab (
   a int,
   b varchar(50),
   c varchar(50)
);

INSERT INTO tab VALUES (NULL, NULL, NULL);

SET SESSION sql_mode='STRICT_TRANS_TABLES';
SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab ADD PRIMARY KEY (a);
ERROR 1265 (01000): Data truncated for column 'a' at row 1

And this fails:

CREATE OR REPLACE TABLE tab (
   a int,
   b varchar(50),
   c varchar(50)
);

INSERT INTO tab VALUES (1, NULL, NULL);
INSERT INTO tab VALUES (1, NULL, NULL);

SET SESSION sql_mode='STRICT_TRANS_TABLES';
SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab ADD PRIMARY KEY (a);
ERROR 1062 (23000): Duplicate entry '1' for key 'PRIMARY'

This applies to ALTER TABLE ... ADD PRIMARY KEY for InnoDB tables.

ALTER TABLE ... DROP PRIMARY KEY

InnoDB does not support dropping a primary key with ALGORITHM set to INPLACE in most cases.

If you try to do so, then you will see an error. InnoDB only supports this operation with ALGORITHM set to COPY. Concurrent DML is *not* permitted.

However, there is an exception. If you are dropping a primary key, and adding a new one at the same time, then that operation can be performed with ALGORITHM set to INPLACE. This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab DROP PRIMARY KEY;
ERROR 1846 (0A000): ALGORITHM=INPLACE is not supported. Reason: Dropping a primary key is not allowed without also adding a new primary key. Try ALGORITHM=COPY

But this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION sql_mode='STRICT_TRANS_TABLES';
SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab DROP PRIMARY KEY, ADD PRIMARY KEY (b);
Query OK, 0 rows affected (0.020 sec)

This applies to ALTER TABLE ... DROP PRIMARY KEY for InnoDB tables.

ALTER TABLE ... ADD INDEX and CREATE INDEX

InnoDB supports adding an index to a table with ALGORITHM set to INPLACE. The table is not rebuilt. This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

However, there is an exception. Adding a FULLTEXT index to a table that does not have a user-defined FTS_DOC_ID column will require the table to be rebuilt once. When the table is rebuilt, the system adds a hidden FTS_DOC_ID column. From that point forward, adding additional FULLTEXT indexes to the same table will not require the table to be rebuilt. In this scenario, this operation supports a read-only locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to SHARED. When this strategy is used, read-only concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab ADD INDEX b_index (b);
Query OK, 0 rows affected (0.010 sec)

And this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
CREATE INDEX b_index ON tab (b);
Query OK, 0 rows affected (0.011 sec)

And this succeeds, but requires the table to be rebuilt:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab ADD FULLTEXT INDEX b_index (b);
Query OK, 0 rows affected (0.055 sec)

This applies to ALTER TABLE ... ADD INDEX and CREATE INDEX for InnoDB tables.

ALTER TABLE ... DROP INDEX and DROP INDEX

InnoDB supports dropping indexes from a table with ALGORITHM set to INPLACE.

This operation only changes the table’s metadata, so the table does not have to be rebuilt.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   INDEX b_index (b)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab DROP INDEX b_index;

And this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   INDEX b_index (b)
);

SET SESSION alter_algorithm='INPLACE';
DROP INDEX b_index ON tab;

This applies to ALTER TABLE ... DROP INDEX and DROP INDEX for InnoDB tables.

ALTER TABLE ... ADD FOREIGN KEY

InnoDB supports adding foreign key constraints to a table with ALGORITHM set to INPLACE. In order to add a new foreign key constraint to a table with ALGORITHM set to INPLACE, the foreign_key_checks system variable needs to be set to OFF. If it is set to ON, then ALGORITHM=COPY is required.

This operation only changes the table’s metadata, so the table does not have to be rebuilt.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this fails:

CREATE OR REPLACE TABLE tab1 (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   d int
);

CREATE OR REPLACE TABLE tab2 (
   a int PRIMARY KEY,
   b varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab1 ADD FOREIGN KEY tab2_fk (d) REFERENCES tab2 (a);
ERROR 1846 (0A000): ALGORITHM=INPLACE is not supported. Reason: Adding foreign keys needs foreign_key_checks=OFF. Try ALGORITHM=COPY

But this succeeds:

CREATE OR REPLACE TABLE tab1 (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   d int
);

CREATE OR REPLACE TABLE tab2 (
   a int PRIMARY KEY,
   b varchar(50)
);

SET SESSION foreign_key_checks=OFF;
SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab1 ADD FOREIGN KEY tab2_fk (d) REFERENCES tab2 (a);
Query OK, 0 rows affected (0.011 sec)

This applies to ALTER TABLE ... ADD FOREIGN KEY for InnoDB tables.

ALTER TABLE ... DROP FOREIGN KEY

InnoDB supports dropping foreign key constraints from a table with ALGORITHM set to INPLACE.

This operation only changes the table’s metadata, so the table does not have to be rebuilt.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab2 (
   a int PRIMARY KEY,
   b varchar(50)
);

CREATE OR REPLACE TABLE tab1 (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   d int,
   FOREIGN KEY tab2_fk (d) REFERENCES tab2 (a)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab1 DROP FOREIGN KEY tab2_fk;
Query OK, 0 rows affected (0.005 sec)

This applies to ALTER TABLE ... DROP FOREIGN KEY for InnoDB tables.

Table Operations

ALTER TABLE ... AUTO_INCREMENT=...

InnoDB supports changing a table’s AUTO_INCREMENT value with ALGORITHM set to INPLACE. This operation should finish instantly. The table is not rebuilt.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab AUTO_INCREMENT=100;
Query OK, 0 rows affected (0.004 sec)

This applies to ALTER TABLE ... AUTO_INCREMENT=... for InnoDB tables.

ALTER TABLE ... ROW_FORMAT=...

InnoDB supports changing a table’s row format with ALGORITHM set to INPLACE.

The table is rebuilt, which means that all of the data is reorganized substantially, and the indexes are rebuilt. As a result, the operation is quite expensive.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) ROW_FORMAT=DYNAMIC;

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab ROW_FORMAT=COMPRESSED;
Query OK, 0 rows affected (0.025 sec)

This applies to ALTER TABLE ... ROW_FORMAT=... for InnoDB tables.

ALTER TABLE ... KEY_BLOCK_SIZE=...

InnoDB supports changing a table’s KEY_BLOCK_SIZE with ALGORITHM set to INPLACE.

The table is rebuilt, which means that all of the data is reorganized substantially, and the indexes are rebuilt. As a result, the operation is quite expensive.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) ROW_FORMAT=COMPRESSED
  KEY_BLOCK_SIZE=4;

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab KEY_BLOCK_SIZE=2;
Query OK, 0 rows affected (0.021 sec)

This applies to KEY_BLOCK_SIZE=... for InnoDB tables.

ALTER TABLE ... PAGE_COMPRESSED=... and ALTER TABLE ... PAGE_COMPRESSION_LEVEL=...

In MariaDB 10.3.10 and later, InnoDB supports setting a table’s PAGE_COMPRESSED value to 1 with ALGORITHM set to INPLACE. InnoDB also supports changing a table’s PAGE_COMPRESSED value from 1 to 0 with ALGORITHM set to INPLACE.

In these versions, InnoDB also supports changing a table’s PAGE_COMPRESSION_LEVEL value with ALGORITHM set to INPLACE.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

See MDEV-16328 for more information.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab PAGE_COMPRESSED=1;
Query OK, 0 rows affected (0.006 sec)

And this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) PAGE_COMPRESSED=1;

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab PAGE_COMPRESSED=0;
Query OK, 0 rows affected (0.020 sec)

And this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) PAGE_COMPRESSED=1
  PAGE_COMPRESSION_LEVEL=5;

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab PAGE_COMPRESSION_LEVEL=4;
Query OK, 0 rows affected (0.006 sec)

This applies to PAGE_COMPRESSED=... and PAGE_COMPRESSION_LEVEL=... for InnoDB tables.

ALTER TABLE ... DROP SYSTEM VERSIONING

InnoDB supports dropping system versioning from a table with ALGORITHM set to INPLACE.

This operation supports the read-only locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to SHARED. When this strategy is used, read-only concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) WITH SYSTEM VERSIONING;

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab DROP SYSTEM VERSIONING;

This applies to ALTER TABLE ... DROP SYSTEM VERSIONING for InnoDB tables.

ALTER TABLE ... DROP CONSTRAINT

In MariaDB 10.3.6 and later, InnoDB supports dropping a CHECK constraint from a table with ALGORITHM set to INPLACE. See MDEV-16331 for more information.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   CONSTRAINT b_not_empty CHECK (b != '')
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab DROP CONSTRAINT b_not_empty;
Query OK, 0 rows affected (0.004 sec)

This applies to ALTER TABLE ... DROP CONSTRAINT for InnoDB tables.

ALTER TABLE ... FORCE

InnoDB supports forcing a table rebuild with ALGORITHM set to INPLACE.

The table is rebuilt, which means that all of the data is reorganized substantially, and the indexes are rebuilt. As a result, the operation is quite expensive.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab FORCE;
Query OK, 0 rows affected (0.022 sec)

This applies to ALTER TABLE ... FORCE for InnoDB tables.

ALTER TABLE ... ENGINE=InnoDB

InnoDB supports forcing a table rebuild with ALGORITHM set to INPLACE.

The table is rebuilt, which means that all of the data is reorganized substantially, and the indexes are rebuilt. As a result, the operation is quite expensive.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab ENGINE=InnoDB;
Query OK, 0 rows affected (0.022 sec)

This applies to ALTER TABLE ... ENGINE=InnoDB for InnoDB tables.

OPTIMIZE TABLE ...

InnoDB supports optimizing a table with with ALGORITHM set to INPLACE.

If the innodb_defragment system variable is set to OFF, and if the innodb_optimize_fulltext_only system variable is also set to OFF, then OPTIMIZE TABLE will be equivalent to ALTER TABLE … FORCE.

The table is rebuilt, which means that all of the data is reorganized substantially, and the indexes are rebuilt. As a result, the operation is quite expensive.

If either of the previously mentioned system variables is set to ON, then OPTIMIZE TABLE will optimize some data without rebuilding the table. However, the file size will not be reduced.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SHOW GLOBAL VARIABLES WHERE Variable_name IN('innodb_defragment', 'innodb_optimize_fulltext_only');
+-------------------------------+-------+
| Variable_name                 | Value |
+-------------------------------+-------+
| innodb_defragment             | OFF   |
| innodb_optimize_fulltext_only | OFF   |
+-------------------------------+-------+

SET SESSION alter_algorithm='INPLACE';
OPTIMIZE TABLE tab;
+---------+----------+----------+-------------------------------------------------------------------+
| Table   | Op       | Msg_type | Msg_text                                                          |
+---------+----------+----------+-------------------------------------------------------------------+
| db1.tab | optimize | note     | Table does not support optimize, doing recreate + analyze instead |
| db1.tab | optimize | status   | OK                                                                |
+---------+----------+----------+-------------------------------------------------------------------+
2 rows in set (0.026 sec)

And this succeeds, but the table is not rebuilt:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET GLOBAL innodb_defragment=ON;
SHOW GLOBAL VARIABLES WHERE Variable_name IN('innodb_defragment', 'innodb_optimize_fulltext_only');
+-------------------------------+-------+
| Variable_name                 | Value |
+-------------------------------+-------+
| innodb_defragment             | ON    |
| innodb_optimize_fulltext_only | OFF   |
+-------------------------------+-------+

SET SESSION alter_algorithm='INPLACE';
OPTIMIZE TABLE tab;
+---------+----------+----------+----------+
| Table   | Op       | Msg_type | Msg_text |
+---------+----------+----------+----------+
| db1.tab | optimize | status   | OK       |
+---------+----------+----------+----------+
1 row in set (0.004 sec)

This applies to OPTIMIZE TABLE for InnoDB tables.

ALTER TABLE ... RENAME TO and RENAME TABLE ...

InnoDB supports renaming a table with ALGORITHM set to INPLACE.

This operation only changes the table’s metadata, so the table does not have to be rebuilt.

This operation supports the exclusive locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to EXCLUSIVE. When this strategy is used, concurrent DML is not permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
ALTER TABLE tab RENAME TO old_tab;
Query OK, 0 rows affected (0.011 sec)

And this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INPLACE';
RENAME TABLE tab TO old_tab;

This applies to ALTER TABLE ... RENAME TO and RENAME TABLE for InnoDB tables.

InnoDB Online DDL Operations with ALGORITHM=NOCOPY

$
0
0

Feed: MariaDB Knowledge Base Article Feed.
Author: .

Column Operations

ALTER TABLE ... ADD COLUMN

In MariaDB 10.3.2 and later, InnoDB supports adding columns to a table with ALGORITHM set to NOCOPY if the new column is the last column in the table. See MDEV-11369 for more information. If the table has a hidden FTS_DOC_ID column is present, then this is not supported.

In MariaDB 10.4 and later, InnoDB supports adding columns to a table with ALGORITHM set to NOCOPY, regardless of where in the column list the new column is added.

With the exception of adding an auto-increment column, this operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab ADD COLUMN c varchar(50);
Query OK, 0 rows affected (0.004 sec)

And this succeeds in MariaDB 10.4 and later:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab ADD COLUMN c varchar(50) AFTER a;
Query OK, 0 rows affected (0.004 sec)

This applies to ALTER TABLE ... ADD COLUMN for InnoDB tables.

ALTER TABLE ... DROP COLUMN

In MariaDB 10.4 and later, InnoDB supports dropping columns from a table with ALGORITHM set to NOCOPY. See MDEV-15562 for more information.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab DROP COLUMN c;
Query OK, 0 rows affected (0.004 sec)

This applies to ALTER TABLE ... DROP COLUMN for InnoDB tables.

ALTER TABLE ... MODIFY COLUMN

This applies to ALTER TABLE ... MODIFY COLUMN for InnoDB tables.

Reordering Columns

In MariaDB 10.4 and later, InnoDB supports reordering columns within a table with ALGORITHM set to NOCOPY. See MDEV-15562 for more information.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab MODIFY COLUMN c varchar(50) AFTER a;
Query OK, 0 rows affected (0.004 sec)

Changing the Data Type of a Column

InnoDB does not support modifying a column’s data type with ALGORITHM set to NOCOPY in most cases. There are a couple exceptions:

  • In MariaDB 10.4.3 and later, InnoDB supports converting a column from VARCHAR to CHAR, or from VARBINARY to BINARY, or from one integer type to another integer type with ALGORITHM set to NOCOPY, as long as the size is greater than or equal to the original size. See MDEV-15563 for more information.

The supported operations in this category support the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab MODIFY COLUMN c int;
ERROR 1846 (0A000): ALGORITHM=NOCOPY is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY

But this succeeds in MariaDB 10.4.3 and later:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab MODIFY COLUMN c varchar(100);
Query OK, 0 rows affected (0.004 sec)

Changing a Column to NULL

In MariaDB 10.4.3 and later, InnoDB supports modifying a column to allow NULL values with ALGORITHM set to NOCOPY if the ROW_FORMAT table option is set to REDUNDANT. See MDEV-15563 for more information.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50) NOT NULL
) ROW_FORMAT=REDUNDANT;

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab MODIFY COLUMN c varchar(50) NULL;
Query OK, 0 rows affected (0.004 sec)

Changing a Column to NOT NULL

InnoDB does not support modifying a column to not allow NULL values with ALGORITHM set to NOCOPY.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) ROW_FORMAT=REDUNDANT;

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab MODIFY COLUMN c varchar(50) NOT NULL;
ERROR 1845 (0A000): ALGORITHM=NOCOPY is not supported for this operation. Try ALGORITHM=INPLACE

Adding a New ENUM Option

InnoDB supports adding a new ENUM option to a column with ALGORITHM set to NOCOPY. In order to add a new ENUM option with ALGORITHM set to NOCOPY, the following requirements must be met:

  • It must be added to the end of the list.
  • The storage requirements must not change.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c ENUM('red', 'green')
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab MODIFY COLUMN c ENUM('red', 'green', 'blue');
Query OK, 0 rows affected (0.002 sec)

But this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c ENUM('red', 'green')
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab MODIFY COLUMN c ENUM('red', 'blue', 'green');
ERROR 1846 (0A000): ALGORITHM=NOCOPY is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY

Adding a New SET Option

InnoDB supports adding a new SET option to a column with ALGORITHM set to NOCOPY. In order to add a new SET option with ALGORITHM set to NOCOPY, the following requirements must be met:

  • It must be added to the end of the list.
  • The storage requirements must not change.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c SET('red', 'green')
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab MODIFY COLUMN c SET('red', 'green', 'blue');
Query OK, 0 rows affected (0.002 sec)

But this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c SET('red', 'green')
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab MODIFY COLUMN c SET('red', 'blue', 'green');
ERROR 1846 (0A000): ALGORITHM=NOCOPY is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY

Removing System Versioning from a Column

In MariaDB 10.3.8 and later, InnoDB supports removing system versioning from a column with ALGORITHM set to NOCOPY. In order for this to work, the system_versioning_alter_history system variable must be set to KEEP. See MDEV-16330 for more information.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50) WITH SYSTEM VERSIONING
);

SET SESSION system_versioning_alter_history='KEEP';
SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab MODIFY COLUMN c varchar(50) WITHOUT SYSTEM VERSIONING;
Query OK, 0 rows affected (0.004 sec)

ALTER TABLE ... ALTER COLUMN

This applies to ALTER TABLE ... ALTER COLUMN for InnoDB tables.

Setting a Column’s Default Value

InnoDB supports modifying a column’s DEFAULT value with ALGORITHM set to NOCOPY.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab ALTER COLUMN c SET DEFAULT 'No value explicitly provided.';
Query OK, 0 rows affected (0.003 sec)

Removing a Column’s Default Value

InnoDB supports removing a column’s DEFAULT value with ALGORITHM set to NOCOPY.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50) DEFAULT 'No value explicitly provided.'
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab ALTER COLUMN c DROP DEFAULT;
Query OK, 0 rows affected (0.002 sec)

ALTER TABLE ... CHANGE COLUMN

InnoDB supports renaming a column with ALGORITHM set to NOCOPY, unless the column’s data type or attributes changed in addition to the name.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab CHANGE COLUMN c str varchar(50);
Query OK, 0 rows affected (0.004 sec)

But this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab CHANGE COLUMN c num int;
ERROR 1846 (0A000): ALGORITHM=NOCOPY is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY

This applies to ALTER TABLE ... CHANGE COLUMN for InnoDB tables.

Generated Columns

Generated columns do not currently support online DDL for all of the same operations that are supported for “real” columns.

See Generated (Virtual and Persistent/Stored) Columns: Statement Support for more information on the limitations.

Index Operations

ALTER TABLE ... ADD PRIMARY KEY

InnoDB does not support adding a primary key to a table with ALGORITHM set to NOCOPY.

For example:

CREATE OR REPLACE TABLE tab (
   a int,
   b varchar(50),
   c varchar(50)
);

SET SESSION sql_mode='STRICT_TRANS_TABLES';
SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab ADD PRIMARY KEY (a);
ERROR 1845 (0A000): ALGORITHM=NOCOPY is not supported for this operation. Try ALGORITHM=INPLACE

This applies to ALTER TABLE ... ADD PRIMARY KEY for InnoDB tables.

ALTER TABLE ... DROP PRIMARY KEY

InnoDB does not support dropping a primary key with ALGORITHM set to NOCOPY.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab DROP PRIMARY KEY;
ERROR 1846 (0A000): ALGORITHM=NOCOPY is not supported. Reason: Dropping a primary key is not allowed without also adding a new primary key. Try ALGORITHM=COPY

This applies to ALTER TABLE ... DROP PRIMARY KEY for InnoDB tables.

ALTER TABLE ... ADD INDEX and CREATE INDEX

InnoDB supports adding an index to a table with ALGORITHM set to NOCOPY.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab ADD INDEX b_index (b);
Query OK, 0 rows affected (0.009 sec)

And this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
CREATE INDEX b_index ON tab (b);
Query OK, 0 rows affected (0.009 sec)

This applies to ALTER TABLE ... ADD INDEX and CREATE INDEX for InnoDB tables.

ALTER TABLE ... DROP INDEX and DROP INDEX

InnoDB supports dropping indexes from a table with ALGORITHM set to NOCOPY.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   INDEX b_index (b)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab DROP INDEX b_index;
Query OK, 0 rows affected (0.008 sec)

And this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   INDEX b_index (b)
);

SET SESSION alter_algorithm='NOCOPY';
DROP INDEX b_index ON tab;
Query OK, 0 rows affected (0.007 sec)

This applies to ALTER TABLE ... DROP INDEX and DROP INDEX for InnoDB tables.

ALTER TABLE ... ADD FOREIGN KEY

InnoDB does supports adding foreign key constraints to a table with ALGORITHM set to NOCOPY. In order to add a new foreign key constraint to a table with ALGORITHM set to NOCOPY, the foreign_key_checks system variable needs to be set to OFF. If it is set to ON, then ALGORITHM=COPY is required.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this fails:

CREATE OR REPLACE TABLE tab1 (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   d int
);

CREATE OR REPLACE TABLE tab2 (
   a int PRIMARY KEY,
   b varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab1 ADD FOREIGN KEY tab2_fk (d) REFERENCES tab2 (a);
ERROR 1846 (0A000): ALGORITHM=NOCOPY is not supported. Reason: Adding foreign keys needs foreign_key_checks=OFF. Try ALGORITHM=COPY

But this succeeds:

CREATE OR REPLACE TABLE tab1 (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   d int
);

CREATE OR REPLACE TABLE tab2 (
   a int PRIMARY KEY,
   b varchar(50)
);

SET SESSION foreign_key_checks=OFF;
SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab1 ADD FOREIGN KEY tab2_fk (d) REFERENCES tab2 (a);
Query OK, 0 rows affected (0.011 sec)

This applies to ALTER TABLE ... ADD FOREIGN KEY for InnoDB tables.

ALTER TABLE ... DROP FOREIGN KEY

InnoDB supports dropping foreign key constraints from a table with ALGORITHM set to NOCOPY.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab2 (
   a int PRIMARY KEY,
   b varchar(50)
);

CREATE OR REPLACE TABLE tab1 (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   d int,
   FOREIGN KEY tab2_fk (d) REFERENCES tab2 (a)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab1 DROP FOREIGN KEY tab2_fk; 
Query OK, 0 rows affected (0.004 sec)

This applies to ALTER TABLE ... DROP FOREIGN KEY for InnoDB tables.

Table Operations

ALTER TABLE ... AUTO_INCREMENT=...

InnoDB supports changing a table’s AUTO_INCREMENT value with ALGORITHM set to NOCOPY.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab AUTO_INCREMENT=100;
Query OK, 0 rows affected (0.002 sec)

This applies to ALTER TABLE ... AUTO_INCREMENT=... for InnoDB tables.

ALTER TABLE ... ROW_FORMAT=...

InnoDB does not support changing a table’s row format with ALGORITHM set to NOCOPY.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) ROW_FORMAT=DYNAMIC;

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab ROW_FORMAT=COMPRESSED;
ERROR 1846 (0A000): ALGORITHM=NOCOPY is not supported. Reason: Changing table options requires the table to be rebuilt. Try ALGORITHM=INPLACE

This applies to ALTER TABLE ... ROW_FORMAT=... for InnoDB tables.

ALTER TABLE ... KEY_BLOCK_SIZE=...

InnoDB does not support changing a table’s KEY_BLOCK_SIZE with ALGORITHM set to NOCOPY.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) ROW_FORMAT=COMPRESSED
  KEY_BLOCK_SIZE=4;

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab KEY_BLOCK_SIZE=2;
ERROR 1846 (0A000): ALGORITHM=NOCOPY is not supported. Reason: Changing table options requires the table to be rebuilt. Try ALGORITHM=INPLACE

This applies to KEY_BLOCK_SIZE=... for InnoDB tables.

ALTER TABLE ... PAGE_COMPRESSED=1 and ALTER TABLE ... PAGE_COMPRESSION_LEVEL=...

In MariaDB 10.3.10 and later, InnoDB supports setting a table’s PAGE_COMPRESSED value to 1 with ALGORITHM set to NOCOPY. InnoDB does not support changing a table’s PAGE_COMPRESSED value from 1 to 0 with ALGORITHM set to NOCOPY.

In these versions, InnoDB also supports changing a table’s PAGE_COMPRESSION_LEVEL value with ALGORITHM set to NOCOPY.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

See MDEV-16328 for more information.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab PAGE_COMPRESSED=1;
Query OK, 0 rows affected (0.004 sec)

And this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) PAGE_COMPRESSED=1
  PAGE_COMPRESSION_LEVEL=5;

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab PAGE_COMPRESSION_LEVEL=4;
Query OK, 0 rows affected (0.004 sec)

But this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) PAGE_COMPRESSED=1;

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab PAGE_COMPRESSED=0;
ERROR 1846 (0A000): ALGORITHM=NOCOPY is not supported. Reason: Changing table options requires the table to be rebuilt. Try ALGORITHM=INPLACE

This applies to ALTER TABLE ... PAGE_COMPRESSED=... and ALTER TABLE ... PAGE_COMPRESSION_LEVEL=... for InnoDB tables.

ALTER TABLE ... DROP SYSTEM VERSIONING

InnoDB does not support dropping system versioning from a table with ALGORITHM set to NOCOPY.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) WITH SYSTEM VERSIONING;

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab DROP SYSTEM VERSIONING;
ERROR 1845 (0A000): ALGORITHM=NOCOPY is not supported for this operation. Try ALGORITHM=INPLACE

This applies to ALTER TABLE ... DROP SYSTEM VERSIONING for InnoDB tables.

ALTER TABLE ... DROP CONSTRAINT

In MariaDB 10.3.6 and later, InnoDB supports dropping a CHECK constraint from a table with ALGORITHM set to NOCOPY. See MDEV-16331 for more information.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   CONSTRAINT b_not_empty CHECK (b != '')
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab DROP CONSTRAINT b_not_empty;
Query OK, 0 rows affected (0.002 sec)

This applies to ALTER TABLE ... DROP CONSTRAINT for InnoDB tables.

ALTER TABLE ... FORCE

InnoDB does not support forcing a table rebuild with ALGORITHM set to NOCOPY.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab FORCE;
ERROR 1845 (0A000): ALGORITHM=NOCOPY is not supported for this operation. Try ALGORITHM=INPLACE

This applies to ALTER TABLE ... FORCE for InnoDB tables.

ALTER TABLE ... ENGINE=InnoDB

InnoDB does not support forcing a table rebuild with ALGORITHM set to NOCOPY.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab ENGINE=InnoDB;
ERROR 1845 (0A000): ALGORITHM=NOCOPY is not supported for this operation. Try ALGORITHM=INPLACE

This applies to ALTER TABLE ... ENGINE=InnoDB for InnoDB tables.

OPTIMIZE TABLE ...

InnoDB does not support optimizing a table with with ALGORITHM set to NOCOPY.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SHOW GLOBAL VARIABLES WHERE Variable_name IN('innodb_defragment', 'innodb_optimize_fulltext_only');
+-------------------------------+-------+
| Variable_name                 | Value |
+-------------------------------+-------+
| innodb_defragment             | OFF   |
| innodb_optimize_fulltext_only | OFF   |
+-------------------------------+-------+
2 rows in set (0.001 sec)

SET SESSION alter_algorithm='NOCOPY';
OPTIMIZE TABLE tab;
+---------+----------+----------+-----------------------------------------------------------------------------+
| Table   | Op       | Msg_type | Msg_text                                                                    |
+---------+----------+----------+-----------------------------------------------------------------------------+
| db1.tab | optimize | note     | Table does not support optimize, doing recreate + analyze instead           |
| db1.tab | optimize | error    | ALGORITHM=NOCOPY is not supported for this operation. Try ALGORITHM=INPLACE |
| db1.tab | optimize | status   | Operation failed                                                            |
+---------+----------+----------+-----------------------------------------------------------------------------+
3 rows in set, 1 warning (0.002 sec)

This applies to OPTIMIZE TABLE for InnoDB tables.

ALTER TABLE ... RENAME TO and RENAME TABLE ...

InnoDB supports renaming a table with ALGORITHM set to NOCOPY.

This operation supports the exclusive locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to EXCLUSIVE. When this strategy is used, concurrent DML is not permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
ALTER TABLE tab RENAME TO old_tab;
Query OK, 0 rows affected (0.008 sec)

And this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='NOCOPY';
RENAME TABLE tab TO old_tab;
Query OK, 0 rows affected (0.008 sec)

This applies to ALTER TABLE ... RENAME TO and RENAME TABLE for InnoDB tables.

InnoDB Online DDL Operations with ALGORITHM=INSTANT

$
0
0

Feed: MariaDB Knowledge Base Article Feed.
Author: .

Column Operations

ALTER TABLE ... ADD COLUMN

In MariaDB 10.3.2 and later, InnoDB supports adding columns to a table with ALGORITHM set to INSTANT if the new column is the last column in the table. See MDEV-11369 for more information. If the table has a hidden FTS_DOC_ID column is present, then this is not supported.

In MariaDB 10.4 and later, InnoDB supports adding columns to a table with ALGORITHM set to INSTANT, regardless of where in the column list the new column is added.

With the exception of adding an auto-increment column, this operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab ADD COLUMN c varchar(50);
Query OK, 0 rows affected (0.004 sec)

And this succeeds in MariaDB 10.4 and later:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab ADD COLUMN c varchar(50) AFTER a;
Query OK, 0 rows affected (0.004 sec)

This applies to ALTER TABLE ... ADD COLUMN for InnoDB tables.

See Instant ADD COLUMN for InnoDB for more information.

ALTER TABLE ... DROP COLUMN

In MariaDB 10.4 and later, InnoDB supports dropping columns from a table with ALGORITHM set to INSTANT. See MDEV-15562 for more information.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab DROP COLUMN c;
Query OK, 0 rows affected (0.004 sec)

This applies to ALTER TABLE ... DROP COLUMN for InnoDB tables.

ALTER TABLE ... MODIFY COLUMN

This applies to ALTER TABLE ... MODIFY COLUMN for InnoDB tables.

Reordering Columns

In MariaDB 10.4 and later, InnoDB supports reordering columns within a table with ALGORITHM set to INSTANT. See MDEV-15562 for more information.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab MODIFY COLUMN c varchar(50) AFTER a;
Query OK, 0 rows affected (0.004 sec)

Changing the Data Type of a Column

InnoDB does not support modifying a column’s data type with ALGORITHM set to INSTANT in most cases. There are a couple exceptions:

  • In MariaDB 10.4.3 and later, InnoDB supports converting a column from VARCHAR to CHAR, or from VARBINARY to BINARY, or from one integer type to another integer type with ALGORITHM set to INSTANT, as long as the size is greater than or equal to the original size. See MDEV-15563 for more information.

The supported operations in this category support the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab MODIFY COLUMN c int;
ERROR 1846 (0A000): ALGORITHM=INSTANT is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY

But this succeeds in MariaDB 10.4.3 and later:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab MODIFY COLUMN c varchar(100);
Query OK, 0 rows affected (0.004 sec)

Changing a Column to NULL

In MariaDB 10.4.3 and later, InnoDB supports modifying a column to allow NULL values with ALGORITHM set to INSTANT if the ROW_FORMAT table option is set to REDUNDANT. See MDEV-15563 for more information.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50) NOT NULL
) ROW_FORMAT=REDUNDANT;

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab MODIFY COLUMN c varchar(50) NULL;
Query OK, 0 rows affected (0.004 sec)

Changing a Column to NOT NULL

InnoDB does not support modifying a column to not allow NULL values with ALGORITHM set to INSTANT.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) ROW_FORMAT=REDUNDANT;

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab MODIFY COLUMN c varchar(50) NOT NULL;
ERROR 1845 (0A000): ALGORITHM=INSTANT is not supported for this operation. Try ALGORITHM=INPLACE

Adding a New ENUM Option

InnoDB supports adding a new ENUM option to a column with ALGORITHM set to INSTANT. In order to add a new ENUM option with ALGORITHM set to INSTANT, the following requirements must be met:

  • It must be added to the end of the list.
  • The storage requirements must not change.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c ENUM('red', 'green')
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab MODIFY COLUMN c ENUM('red', 'green', 'blue');
Query OK, 0 rows affected (0.002 sec)

But this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c ENUM('red', 'green')
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab MODIFY COLUMN c ENUM('red', 'blue', 'green');
ERROR 1846 (0A000): ALGORITHM=INSTANT is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY

Adding a New SET Option

InnoDB supports adding a new SET option to a column with ALGORITHM set to INSTANT. In order to add a new SET option with ALGORITHM set to INSTANT, the following requirements must be met:

  • It must be added to the end of the list.
  • The storage requirements must not change.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c SET('red', 'green')
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab MODIFY COLUMN c SET('red', 'green', 'blue');
Query OK, 0 rows affected (0.002 sec)

But this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c SET('red', 'green')
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab MODIFY COLUMN c SET('red', 'blue', 'green');
ERROR 1846 (0A000): ALGORITHM=INSTANT is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY

Removing System Versioning from a Column

In MariaDB 10.3.8 and later, InnoDB supports removing system versioning from a column with ALGORITHM set to INSTANT. In order for this to work, the system_versioning_alter_history system variable must be set to KEEP. See MDEV-16330 for more information.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50) WITH SYSTEM VERSIONING
);

SET SESSION system_versioning_alter_history='KEEP';
SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab MODIFY COLUMN c varchar(50) WITHOUT SYSTEM VERSIONING;
Query OK, 0 rows affected (0.004 sec)

ALTER TABLE ... ALTER COLUMN

This applies to ALTER TABLE ... ALTER COLUMN for InnoDB tables.

Setting a Column’s Default Value

InnoDB supports modifying a column’s DEFAULT value with ALGORITHM set to INSTANT.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab ALTER COLUMN c SET DEFAULT 'No value explicitly provided.';
Query OK, 0 rows affected (0.003 sec)

Removing a Column’s Default Value

InnoDB supports removing a column’s DEFAULT value with ALGORITHM set to INSTANT.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50) DEFAULT 'No value explicitly provided.'
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab ALTER COLUMN c DROP DEFAULT;
Query OK, 0 rows affected (0.002 sec)

ALTER TABLE ... CHANGE COLUMN

InnoDB supports renaming a column with ALGORITHM set to INSTANT, unless the column’s data type or attributes changed in addition to the name.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab CHANGE COLUMN c str varchar(50);
Query OK, 0 rows affected (0.004 sec)

But this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab CHANGE COLUMN c num int;
ERROR 1846 (0A000): ALGORITHM=INSTANT is not supported. Reason: Cannot change column type INPLACE. Try ALGORITHM=COPY

This applies to ALTER TABLE ... CHANGE COLUMN for InnoDB tables.

Generated Columns

Generated columns do not currently support online DDL for all of the same operations that are supported for “real” columns.

See Generated (Virtual and Persistent/Stored) Columns: Statement Support for more information on the limitations.

Index Operations

ALTER TABLE ... ADD PRIMARY KEY

InnoDB does not support adding a primary key to a table with ALGORITHM set to INSTANT.

For example:

CREATE OR REPLACE TABLE tab (
   a int,
   b varchar(50),
   c varchar(50)
);

SET SESSION sql_mode='STRICT_TRANS_TABLES';
SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab ADD PRIMARY KEY (a);
ERROR 1845 (0A000): ALGORITHM=INSTANT is not supported for this operation. Try ALGORITHM=INPLACE

This applies to ALTER TABLE ... ADD PRIMARY KEY for InnoDB tables.

ALTER TABLE ... DROP PRIMARY KEY

InnoDB does not support dropping a primary key with ALGORITHM set to INSTANT.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab DROP PRIMARY KEY;
ERROR 1846 (0A000): ALGORITHM=INSTANT is not supported. Reason: Dropping a primary key is not allowed without also adding a new primary key. Try ALGORITHM=COPY

This applies to ALTER TABLE ... DROP PRIMARY KEY for InnoDB tables.

ALTER TABLE ... ADD INDEX and CREATE INDEX

InnoDB does not support adding an index to a table with ALGORITHM set to INSTANT.

For example, this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab ADD INDEX b_index (b);
ERROR 1846 (0A000): ALGORITHM=INSTANT is not supported. Reason: ADD INDEX. Try ALGORITHM=NOCOPY

And this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
CREATE INDEX b_index ON tab (b);
ERROR 1846 (0A000): ALGORITHM=INSTANT is not supported. Reason: ADD INDEX. Try ALGORITHM=NOCOPY

This applies to ALTER TABLE ... ADD INDEX and CREATE INDEX for InnoDB tables.

ALTER TABLE ... DROP INDEX and DROP INDEX

InnoDB supports dropping indexes from a table with ALGORITHM set to INSTANT.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   INDEX b_index (b)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab DROP INDEX b_index;
Query OK, 0 rows affected (0.008 sec)

And this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   INDEX b_index (b)
);

SET SESSION alter_algorithm='INSTANT';
DROP INDEX b_index ON tab;
Query OK, 0 rows affected (0.007 sec)

This applies to ALTER TABLE ... DROP INDEX and DROP INDEX for InnoDB tables.

ALTER TABLE ... ADD FOREIGN KEY

InnoDB does not support adding foreign key constraints to a table with ALGORITHM set to INSTANT.

For example:

CREATE OR REPLACE TABLE tab1 (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   d int
);

CREATE OR REPLACE TABLE tab2 (
   a int PRIMARY KEY,
   b varchar(50)
);

SET SESSION foreign_key_checks=OFF;
SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab1 ADD FOREIGN KEY tab2_fk (d) REFERENCES tab2 (a);
ERROR 1846 (0A000): ALGORITHM=INSTANT is not supported. Reason: ADD INDEX. Try ALGORITHM=NOCOPY

This applies to ALTER TABLE ... ADD FOREIGN KEY for InnoDB tables.

ALTER TABLE ... DROP FOREIGN KEY

InnoDB supports dropping foreign key constraints from a table with ALGORITHM set to INSTANT.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab2 (
   a int PRIMARY KEY,
   b varchar(50)
);

CREATE OR REPLACE TABLE tab1 (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   d int,
   FOREIGN KEY tab2_fk (d) REFERENCES tab2 (a)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab1 DROP FOREIGN KEY tab2_fk; 
Query OK, 0 rows affected (0.004 sec)

This applies to ALTER TABLE ... DROP FOREIGN KEY for InnoDB tables.

Table Operations

ALTER TABLE ... AUTO_INCREMENT=...

InnoDB supports changing a table’s AUTO_INCREMENT value with ALGORITHM set to INSTANT.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab AUTO_INCREMENT=100;
Query OK, 0 rows affected (0.002 sec)

This applies to ALTER TABLE ... AUTO_INCREMENT=... for InnoDB tables.

ALTER TABLE ... ROW_FORMAT=...

InnoDB does not support changing a table’s row format with ALGORITHM set to INSTANT.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) ROW_FORMAT=DYNAMIC;

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab ROW_FORMAT=COMPRESSED;
ERROR 1846 (0A000): ALGORITHM=INSTANT is not supported. Reason: Changing table options requires the table to be rebuilt. Try ALGORITHM=INPLACE

This applies to ALTER TABLE ... ROW_FORMAT=... for InnoDB tables.

ALTER TABLE ... KEY_BLOCK_SIZE=...

InnoDB does not support changing a table’s KEY_BLOCK_SIZE with ALGORITHM set to INSTANT.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) ROW_FORMAT=COMPRESSED
  KEY_BLOCK_SIZE=4;

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab KEY_BLOCK_SIZE=2;
ERROR 1846 (0A000): ALGORITHM=INSTANT is not supported. Reason: Changing table options requires the table to be rebuilt. Try ALGORITHM=INPLACE

This applies to KEY_BLOCK_SIZE=... for InnoDB tables.

ALTER TABLE ... PAGE_COMPRESSED=1 and ALTER TABLE ... PAGE_COMPRESSION_LEVEL=...

In MariaDB 10.3.10 and later, InnoDB supports setting a table’s PAGE_COMPRESSED value to 1 with ALGORITHM set to INSTANT. InnoDB does not support changing a table’s PAGE_COMPRESSED value from 1 to 0 with ALGORITHM set to INSTANT.

In these versions, InnoDB also supports changing a table’s PAGE_COMPRESSION_LEVEL value with ALGORITHM set to INSTANT.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

See MDEV-16328 for more information.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab PAGE_COMPRESSED=1;
Query OK, 0 rows affected (0.004 sec)

And this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) PAGE_COMPRESSED=1
  PAGE_COMPRESSION_LEVEL=5;

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab PAGE_COMPRESSION_LEVEL=4;
Query OK, 0 rows affected (0.004 sec)

But this fails:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) PAGE_COMPRESSED=1;

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab PAGE_COMPRESSED=0;
ERROR 1846 (0A000): ALGORITHM=INSTANT is not supported. Reason: Changing table options requires the table to be rebuilt. Try ALGORITHM=INPLACE

This applies to ALTER TABLE ... PAGE_COMPRESSED=... and ALTER TABLE ... PAGE_COMPRESSION_LEVEL=... for InnoDB tables.

ALTER TABLE ... DROP SYSTEM VERSIONING

InnoDB does not support dropping system versioning from a table with ALGORITHM set to INSTANT.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
) WITH SYSTEM VERSIONING;

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab DROP SYSTEM VERSIONING;
ERROR 1845 (0A000): ALGORITHM=INSTANT is not supported for this operation. Try ALGORITHM=INPLACE

This applies to ALTER TABLE ... DROP SYSTEM VERSIONING for InnoDB tables.

ALTER TABLE ... DROP CONSTRAINT

In MariaDB 10.3.6 and later, InnoDB supports dropping a CHECK constraint from a table with ALGORITHM set to INSTANT. See MDEV-16331 for more information.

This operation supports the non-locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to NONE. When this strategy is used, all concurrent DML is permitted.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50),
   CONSTRAINT b_not_empty CHECK (b != '')
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab DROP CONSTRAINT b_not_empty;
Query OK, 0 rows affected (0.002 sec)

This applies to ALTER TABLE ... DROP CONSTRAINT for InnoDB tables.

ALTER TABLE ... FORCE

InnoDB does not support forcing a table rebuild with ALGORITHM set to INSTANT.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab FORCE;
ERROR 1845 (0A000): ALGORITHM=INSTANT is not supported for this operation. Try ALGORITHM=INPLACE

This applies to ALTER TABLE ... FORCE for InnoDB tables.

ALTER TABLE ... ENGINE=InnoDB

InnoDB does not support forcing a table rebuild with ALGORITHM set to INSTANT.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab ENGINE=InnoDB;
ERROR 1845 (0A000): ALGORITHM=INSTANT is not supported for this operation. Try ALGORITHM=INPLACE

This applies to ALTER TABLE ... ENGINE=InnoDB for InnoDB tables.

OPTIMIZE TABLE ...

InnoDB does not support optimizing a table with with ALGORITHM set to INSTANT.

For example:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SHOW GLOBAL VARIABLES WHERE Variable_name IN('innodb_defragment', 'innodb_optimize_fulltext_only');
+-------------------------------+-------+
| Variable_name                 | Value |
+-------------------------------+-------+
| innodb_defragment             | OFF   |
| innodb_optimize_fulltext_only | OFF   |
+-------------------------------+-------+
2 rows in set (0.001 sec)

SET SESSION alter_algorithm='INSTANT';
OPTIMIZE TABLE tab;
+---------+----------+----------+------------------------------------------------------------------------------+
| Table   | Op       | Msg_type | Msg_text                                                                     |
+---------+----------+----------+------------------------------------------------------------------------------+
| db1.tab | optimize | note     | Table does not support optimize, doing recreate + analyze instead            |
| db1.tab | optimize | error    | ALGORITHM=INSTANT is not supported for this operation. Try ALGORITHM=INPLACE |
| db1.tab | optimize | status   | Operation failed                                                             |
+---------+----------+----------+------------------------------------------------------------------------------+
3 rows in set, 1 warning (0.002 sec)

This applies to OPTIMIZE TABLE for InnoDB tables.

ALTER TABLE ... RENAME TO and RENAME TABLE ...

InnoDB supports renaming a table with ALGORITHM set to INSTANT.

This operation supports the exclusive locking strategy. This strategy can be explicitly chosen by setting the LOCK clause to EXCLUSIVE. When this strategy is used, concurrent DML is not permitted.

For example, this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
ALTER TABLE tab RENAME TO old_tab;
Query OK, 0 rows affected (0.008 sec)

And this succeeds:

CREATE OR REPLACE TABLE tab (
   a int PRIMARY KEY,
   b varchar(50),
   c varchar(50)
);

SET SESSION alter_algorithm='INSTANT';
RENAME TABLE tab TO old_tab;
Query OK, 0 rows affected (0.008 sec)

This applies to ALTER TABLE ... RENAME TO and RENAME TABLE for InnoDB tables.

The MySQL 8.0.16 Maintenance Release is Generally Available

$
0
0

Feed: Planet MySQL
;
Author: Geir Hoydalsvik
;

The MySQL Development team is very happy to announce that MySQL 8.0.16 is now available for download at dev.mysql.com. In addition to bug fixes there are a few new features added in this release.  Please download 8.0.16 from dev.mysql.com or from the MySQL  YumAPT, or SUSE repositories. The source code is available at GitHub. You can find the full list of changes and bug fixes in the 8.0.16 Release Notes. Here are the highlights. Enjoy!

SQL

CHECK constraints (WL#929) This work by Praveenkumar Hulakund implements the CHECK constraint, i.e. the clause [CONSTRAINT [constraint_name]] CHECK (condition) [[NOT] ENFORCED] in CREATE TABLE and ALTER TABLE statements. A check constraint is satisfied if and only if the specified condition evaluates to TRUE or UNKNOWN (for NULL column value) for a row of the table. The constraint is violated otherwise. For example: CREATE TABLE t (s1 INT, CHECK (s1 > 0))  INSERT INTO t VALUES (-1) will fail since the condition is FALSE and INSERT INTO t VALUES (NULL) will succeed since the condition is UNKNOWN. This work implements a commonly requested feature, see for example Bug#3465 and Bug#22759.

Upgrade

Move actions from mysql_upgrade client into mysqld (WL#12413) This work by Amitabh Das moves the actions executed by the MySQL upgrade script to the mysqld itself. The MySQL server now has control of when these actions are executed, and does not rely on the user to remember to run the mysql_upgrade script. I.e. starting with 8.0.16, MySQL will upgrade itself. The goal is to make upgrade simpler and faster by removing the mysql_upgrade step. The mysql_upgrade is deprecated and will be removed in a future MySQL version. Even though mysql_upgrade no longer performs upgrade tasks, it still exits with status 0 (EXIT_SUCCESS) to avoid failures in existing user scripts. See MySQL Upgrade.

GIS

ST_Length with unit  (WL#12657) This work by Torje Digernes implements a second, optional, parameter for ST_Length(ls [, unit]) that names the unit of length to use in the result. Supported units are listed in the Information Schema ST_UNITS_OF_MEASURE table.

Character Sets

Add Chinese collation for utf8mb4  (WL#11825) This work by Xing Zhang adds the Chinese collation for utf8mb4. The collation is named utf8mb4_zh_0900_as_cs, ‘utf8mb4’ is the character set, ‘zh’ is the ISO code for Chinese, ‘0900’ means this collation follows the Unicode standard 9.0, ‘as_cs’ means accent sensitive and case sensitive.

Information Schema

Add metrics for undo truncation (WL#12600) This work by Kevin Lewis adds metrics to the INFORMATION_SCHEMA.INNODB_METRICS table for the background activities during undo tablespace truncation. These metrics are available by SELECT * FROM INFORMATION_SCHEMA.INNODB_METRICS WHERE subsystem = 'undo'; Examples include the number of times the purge thread attempted to truncate undo history and the time the purge thread spent truncating undo history. See WL#12600 for the full list.

Performance Schema

Native Functions  (WL#7803) This work by Chris Powers implements native functions to format or retrieve data from the Performance Schema. The implemented functions are format_pico_time()format_bytes(), PS_current_thread_id(), and  PS_thread_id(). These native functions replace the corresponding stored functions in the MySQL sys schema. The purpose is to reduce overhead by having a C++ implementation rather than a stored function.

Optimizer

Add folding of constants when compared to fields (WL#11935) This work by Dag Wanvik is about internal optimizer improvement. The goal is to speed up execution at the cost of a little more analysis at optimize time. Always true and false comparisons are detected and eliminated. In other cases, the type of the constant is adjusted to match that of the field if they are not the same, avoiding type conversion at execution time.

Subquery optimizations: Make IN optimizations also handle EXISTS  (WL#4389) This work by Roy Lyseng extends the IN semi-join optimizations to handle EXISTS subqueries as well. EXISTS subqueries are handled with the same semi-join strategies as IN, including first-match, materialization, duplicate weedout and loose index scan. In addition, the WHERE condition attached to the subquery is decorrelated into IN-like expression lists, further expanding the possible semi-join strategies. Decorrelation is done for both EXISTS and IN subqueries. The overall goal is to transform more queries into a form where we can do cost based optimizations (e.g. semi-join).

Volcano iterator design (WL#11785 and WL#12074) This work by Steinar H. Gunderson implements the beginning of a new SQL executor. The work is based on the Volcano model, see the original Volcano paper here. The goal of this activity is to simplify the code base, enable new features such as hash join, and enable a better EXPLAIN and EXPLAIN ANALYZE.

Replication

Add Partition Information into the Binary Log (WL#12168) This work by Neha Kumari injects table partitioning information into the binary log. This information will also be available by tools such as mysqlbinlog. Among other possibilities, this extra information can be used to make the replication applier schedule more transactions concurrently.

Add support to binary log encryption key rotation and cleanup (WL#12080) This work by Daogang Qu makes it easier for the user to rotate the binary log master key used to encrypt file passwords of binary and relay log files. For example, as the database owner you might want to rotate binlog encryption master key periodically without the need to restart the server in order to comply with security rules.

Group Replication

MYSQL GCS: Improve XCom Cache Management  (WL#11615) This work by Andre Negrao makes the XCom Cache configurable and dynamic. The user can set the maximum group_replication_message_cache_size for each individual node. In addition, the cache will no longer be bound by a fixed number of entries; instead, it will grow dynamically, so long as the size limit is respected.

Auto-rejoin member to group after an expulsion  (WL#11284) This work by Ricardo Ferreira ensures that expelled nodes can auto-rejoin the group if they are out of it for a reasonably short time. This feature automates rejoin operations for servers that leave the group due to transient communication failures between the peers.

Support Data Fragmentation (WL#11610) This work by Tiago Vale enables packet fragmentation which makes the group communication layer handle large messages much more efficiently, and by doing that avoid false suspicions that could lead to servers being evicted from the group in some cases.

Change exit state action option to READ_ONLY  (WL#12659) This work by Jaideep Karande changes the default value of group_replication_exit_state_action from ABORT_SERVER to READ_ONLY. We decided to change this default back to READ_ONLY based on consistent customer and community feedback.

Tablespace Encryption

Encryption of mysql tablespace  (WL#12063) This work by Mayank Prasad adds functionality to encrypt the MySQL data dictionary which resides in the mysql tablespace. See also  Tablespace Encryption.

Control (enforce and disable) table encryption (WL#12261) This work by Gopal Shankar enables the DBA to have global control over table encryption. With this feature the DBA can grant CREATE TABLE to users while still retaining some control over encryption usage, e.g. enforce that all tables in a certain database are encrypted. The DBA can also make sure people don’t do double encryption in cases where the underlying storage engine is encrypted. See also Tablespace Encryption.

Users and Privileges

MySQL system users  (WL#12098)  This work by Rahul Sisondia adds two new features, the “SYSTEM_USER privilege” and “partial_revokes”. The users granted the ‘SYSTEM_USER’ privilege can be seen as power users, and in effect there will two categories of users, power users and regular users. Regular users can not modify any properties of power users – even if DDL privileges are granted. The purpose of this (optional) feature is to enable a strong separation between those who administrate a hosted environment and those that are using the same environment. The partial_revoke feature simplifies the administration of privileges. With partial_revoke, the DBA can restrict DDL/DML operations on one or more databases even if the user has the required global privileges. Alternatively, the DBA must avoid using global privileges if she wants to protect certain resources, which are more cumbersome and error prone.

Extend GRANT/REVOKE syntax to cover partial revokes information (WL#12820) This work by Harin Vadodaria extends the GRANT/REVOKE syntax with the [AS user [WITH ROLE..]] clause. The purpose is to be able to specify the grantor and its active roles. Normally this does not need to be specified since it is the grantor with its active roles who does the GRANT/REVOKE. However, when the GRANT/REVOKE information is read from the binlog by the  mysqlbinlog tool and is to be replayed by the Server it is replayed by a user who lacks this context. This problem is solved by extending the syntax to cover the missing contextual information.

Separate KILL administration from CONNECTION_ADMIN (WL#12364) This work by Rahul Sisondia ensures that only users with the SYSTEM_USER privilege can kill sessions/queries for other users with the SYSTEM_USER privilege. Without this change, users with CONNECTION_ADMIN and/or SUPER privileges could kill connections belonging to users with the SYSTEM_USER privilege.

Security

Support TLS 1.3 in the server and libmysql  (WL#12361) This work by Ramil Kalimullin adds support for TLS 1.3 in the –tls-version option and enables it by default. MySQL 8.0 with OpenSSL 1.1.1 now supports the TLSv1.0, TLSv1.1, TLSv1.2 and TLSv1.3 protocols.

Allow switching the SSL options for a running server (WL#11541) This work by Georgi Kodinov makes all of the SSL options dynamic by preparing a new SSL context for the listening socket and then replacing the old one. The purpose is to be able to update the certificates without restarting the running server. This work is based on the Facebook contribution found in Bug#90782. See also ALTER INSTANCE RELOAD TLS.

Instrument the keyring into performance schema  (WL#11543) This work by Ivo Roylev adds Performance Schema tables which exposes the metadata of the keys (key IDs, key owner), provided that the user has enough privileges to see these. When using an external key vault it is useful to match the internal key ID used by the keyring infrastructure, to the ID that this key is stored in the key vault. See also the MySQL Keyring.

MySQL Classic Protocol

Add asynchronous, non-blocking C APIs (WL#11381) This work by Bharathy Satish extends the historical C APIs with new non-blocking C APIs. The non blocking version of C APIs does not wait for data to be available on the socket to be read. These APIs will return immediately with a return status. Return status is used to decide if the API has completed its operation or needs to be called again at some later point in time to complete the operation. This work is based on the Facebook contribution found in Bug#89327.

MySQL X Protocol

Error message when connecting with wrong protocol (WL#12240)   This work by Lukasz Kotula ensures that a Classic Protocol Client gets a useful error message (invalid protocol) if it tries to connect to the server at the X Protocol port number.

Session connect attributes (WL#12376)  This work by Tomasz Stepniak implements support for session connection attributes in the X Protocol. Clients can send custom attributes such as the name and version of the application, OS and PID to the server, which can be queried from the performance schema session_connect_attrs table. Such client information can help server administrators to find faulty clients and target potential issues.

Reset connection state  (WL#12375) This work by Grzegorz Szwarc fixes wrong behavior in X Protocol session-reset, session-close, and connection-close messages. The session-reset message will reset a session without reauthenticating or reopening the connection. The session-close message keeps the connection open and but the session must  re-authenticate. The connection-close message will close the current connection.

MySQL Router

HTTP component for Router  (WL#11891, WL#12524, WL#12503) This work by Jan Kneschke adds a tiny HTTP server library to MySQL Router to allow it to expose REST endpoints and a web-interface. The purpose is to ensure observability of the Router e.g. integration with external tools for healthcheck, monitoring, and management purposes. It supports TLS/HTTPS and basic authentication against a secure, file based password storage.

Support for dynamic changes between single and multi master mode  (WL#12089) This work by Pawel Mroszczyk adapts the Router configuration so that it can seamlessly move between single and multi master modes.  The configuration file now has RW and RO Routing sections for all transports regardless of how many PRIMARY or SECONDARY nodes the cluster has.

Allow MySQL Router to log to multiple log sinks (WL#12733) This work by Andrzej Religa allows each log sink to be configured independently. The MySQL Router supports logging to different console, file, syslog (unix), and eventlog (windows). As an example, this work allows the user to log all the messages to file (level=debug) and only errors to the syslog (level=error).

Send log messages to windows eventlog  (WL#9552) This work by Andrzej Religa implements a new eventlog plugin that logs events to the windows event log. Events are logged with the correct event type. The DEBUG level messages are logged as INFORMATION since Windows EventLog does not support DEBUG. Message source for all the messages is “MySQL Router”. In order to correctly display the log source as MySQL Router in the Windows event log, the MySQLRouter binary needs to be added in the Windows registry. For that proper privileges are needed (system admin) when launching the Router for the first time after the eventlog logging has been added to the configuration.

Log rotation on unix-system (WL#8988) This work by Andrzej Religa enables log rotation on demand while the Router is running. The router will reopen the logfile when it receives the SIGHUP signal.

Emphasis output of router executables for better readability  (WL#12604 and WL#12598) This work by Jan Kneschke applies colors and emphasis to  all –help output for the mysqlrouter and the mysqlrouter_* tools as well as the mysqlrouter –bootstrap output. The purpose is to make it easier for humans to read the information output.

Other

Component service interface of my_error API (WL#12544) This work by Murthy Sidagam adds a new component service mysql_runtime_error to allow components to report errors to the client session (if called from a session) or the server error log. It also adds a set of convenience functions to allow easy migration of existing server API symbols to the new service. The service’s default implementation resides in the server component.

Make number of PAUSES in spin loops configurable  (WL#12616) This work by Jakub Lopuszanski introduces a new system variable called innodb_spin_wait_pause_multiplier which defaults to 50 (range 0..100). The purpose is to let the system administrator user adapt the PAUSE to the hardware at hand. For example, Skylake processors have a much slower (~15x) PAUSE instruction than earlier processors.

Pre-parse plugin for DDL statement clause filtering (WL#12668) This work by Sivert Sørumgård implements a plugin which does simple string replacement of DDL statements from dump files based on regular expression matching. This will be an audit type plugin catching and handling pre-parse events. In various situations when restoring output from a dump file, it is relevant to filter out certain clauses from DDL statements. This is useful, e.g. when restoring a dump file into an environment where encryption will not be applied. In such a case, we would like to filter out the encryption clauses from the CREATE TABLE statements.

Add an option to allow server quick settings validation (WL#12360) This work by Nisha Gopalakrishnan introduces a validate-config command line option which allows users to validate their server configuration settings during server start up. The validate-config option will only validate the server options (i.e the output of “–help –verbose”) and does not validate plugin options. The server will exit with an error(1) for first occurrence of invalid configuration. The server will exit with error(0) when it has successfully validated all the configuration options.

Enable use of C++14 in MySQL Server  (WL#12424) This work by Jon Olav Hauglid enables the use of C++14 in the 8.0 version of MySQL server.  C++14 is a small extension over C++11 with mostly bug fixes and minor improvements. Descriptions and code examples of new C++14 features can be found here. This means that the minimal compiler versions are GCC 5.3 (Linux), Clang 4.0 (FreeBSD), XCode 9 (MacOS), Developer Studio 12.6 (Solaris), and Visual Studio 2017 (Windows)

Deprecation and Removals

Deprecate mysql_upgrade client  (WL#12918) This work by Amitabh Das adds a deprecation warning to the mysql_upgrade client in 8.0. The mysql_upgrade is no longer needed since the upgrade code is folded into the server by WL#12413.

Deprecate the non-caching sha256 authentication plugin (WL#12694) This work by Yashwant Sahu deprecates the sha256_password plugin both as a server side authentication plugin and as a client side authentication plugin. Use the default caching_sha2_password instead.  See SHA-256 Pluggable Authentication.

Remove  MyISAM as engines used for on-disk internal temporary tables  (WL#11974) This work by Sergey Gluhov removes the internal_tmp_disk_storage_engine system variable and thus the option to use MyISAM for internal temporary tables. MySQL will now always use the TempTable storage engine introduced in 8.0 for overflow to disk.

Thank you for using MySQL !





MySQL 8.0.16 Check Constraints

$
0
0

Feed: Planet MySQL
;
Author: Dave Stokes
;

Before MySQL 8.0.16 you could put constraint checks into your Data Definition Language (DDL) when creating tables but the server ignored them.  There was much gnashing of teeth as taunts of “It is not a real database” from other databases taunted the MySQL Community. 
 
But with 8.0.16 this has all changed. You can now have your data constraints checked by the server. Below is an example table with two constraints.

mysql>CREATE TABLE parts 
            (id int, cost decimal(5,2) not null check (cost > 0),
             price decimal(5,2) not null check (price > 1.0)
          );
Query OK, 0 rows affected (0.09 sec)

mysql> insert into parts (id,cost,price) values (1,1.10,2.25);
Query OK, 1 row affected (0.03 sec)

mysql> insert into parts (id,cost,price) values (1,1.10,0.25);
ERROR 3819 (HY000): Check constraint ‘parts_chk_2’ is violated.

mysql> insert into parts (id,cost,price) values (2,-1.1,4.25);
ERROR 3819 (HY000): Check constraint ‘parts_chk_1’ is violated.

The first data input above does passes the constraints since both the cost and price columns are greater than the minimum. But not so with the next inserts, So the constraints work!

Finding Out What The Error Message Means

Now seeing Check constraint ‘parts_chk_1’ is violated. tells me the data is out of value. But how do I find out what parts_chk_1 is?

The fist thing to check is the table itself.



mysql> show create table parts;
+——-+———————————————————————————————————————————————————————————————————————————————————————————————+
| Table | Create Table                                                                                                                                                                                                                                                                                |
+——-+———————————————————————————————————————————————————————————————————————————————————————————————+
| parts | CREATE TABLE `parts` (
  `id` int(11) DEFAULT NULL,
  `cost` decimal(5,2) NOT NULL,
  `price` decimal(5,2) NOT NULL,
  CONSTRAINT `parts_chk_1` CHECK ((`cost` > 0)),
  CONSTRAINT `parts_chk_2` CHECK ((`price` > 1.0))

) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci |
+——-+———————————————————————————————————————————————————————————————————————————————————————————————+
1 row in set (0.01 sec)

And you can also see the information in the IS.

mysql> select * from INFORMATION_SCHEMA.CHECK_CONSTRAINTS;
+——————–+——————-+—————–+—————–+
| CONSTRAINT_CATALOG | CONSTRAINT_SCHEMA | CONSTRAINT_NAME | CHECK_CLAUSE    |
+——————–+——————-+—————–+—————–+
| def                | davetest          | xxx_chk_1       | (`age` > 18)    |
| def                | davetest          | parts_chk_1     | (`cost` > 0)    |
| def                | davetest          | parts_chk_2     | (`price` > 1.0) |
| def                | davetest          | client_chk_1    | (`age` > 17)    |
+——————–+——————-+—————–+—————–+
4 rows in set (0.00 sec)

Supercharge cybersecurity with Netflow and Yellowbrick Data

$
0
0

Feed: Yellowbrick Data.
Author: Christian Shrauder
;

Some of the most challenging datasets from a volume and velocity perspective are network related.  Almost every action today, and sometimes no action at all, will generate some kind of network traffic. Common enterprise network data includes IoT, advertising data, PCAP (packet capture), and Netflow. The 2.0 release of the Yellowbrick Data Warehouse includes support for IPv4, IPv6 and MAC addresses. This support enables organizations to take Netflow analysis to a new level.

What is Netflow?

Netflow was developed by Cisco to monitor network traffic. With Netflow, organizations can easily determine where traffic is going and how much traffic is involved from a time or bytes perspective.  Many organizations use Netflow to gain insights about user habits and application use, as well as to identify peak network utilization hours and routes suffering from bottlenecks. Think of Netflow as a solution for viewing network metadata, including source and destination IP addresses, ports, which can be used to determine application use,  session duration, data volume (bytes transferred), and perhaps most importantly, time stamps.

Netflow has gained popularity as enterprises become more introspective to improve cybersecurity. With the exponential growth of tablets and smartphones behind the network perimeter, security scrutiny must move beyond firewalls to the internal network traffic.

The challenge with monitoring and identifying security threats with Netflow is data volume. Every network interaction generates a log line. Multiply this by the number of devices on the network, the applications per device, and the number of users, and it’s easy to see just how much data is being created. Medium-sized businesses generate 10s of millions of rows of data per day, while a multinational enterprise or government agency easily generates trillions of rows per year. Effective cybersecurity hinges on automating risk detection, using predictive analytics to measure the risk of real-time activity against aggregated historical patterns. Anomalous behavior can then be flagged for an investigator to analyze in more detail. The problem is that data volumes are now so massive that historical data is regularly archived to create space for new data, which means that less historical data is being analyzed quickly, which makes the end results less accurate or less timely. Enterprises are struggling for ways to collect Netflow data in near real time and analyze it against billions or trillions of rows of historical data without making a science project out of it. Enter Yellowbrick.

Yellowbrick delivers deeper, faster Netflow insights

Yellowbrick IP and MAC address support optimizes analytics by storing data in ways that enable faster, more effective queries. Let’s take a closer look at how.

Faster, more efficient queries of IP data

Most databases don’t have a specific IP address data type. This leaves users storing the data as a varchar or creating a function to convert the data to an integer.  Varchars are just strings, so they can’t be treated like a number with order or mathematical functions such as < > or ranges in between. To work an IP address to an integer, you have to perform a conversion, which can add significant time when you’re querying millions or billions of rows. The Yellowbrick database treats IP addresses as their own type, similarly to  a number, and supports order and functions. This enables organizations to query ranges with greater than, less than, or between operators without the need to convert data types. With Yellowbrick, you can work with IP addresses in ways similar to the following examples:

or

Enabling rapid data ingest, scanning, and access

Most databases can run a table scan (the most costly activity for most queries) on datasets of one million rows quite quickly. However, as noted above, many Netflow queries now need to scan billions or even trillions of rows. Worse still, it’s increasingly common for multiple large queries to be executing at the same time, resulting in multiple scans at once. Queries can take hours or days to complete, a time lag that is unacceptable for effective cybersecurity.

If you’re reading this, you might already know that Yellowbrick is the world’s first data warehouse designed for flash memory. It can perform table scans at speeds of hundreds of gigabytes per second. A million row table scan is a sub-second activity for Yellowbrick.

Some ways Yellowbrick improves data analytics is by enabling users to define columns for data to be sorted into. Even after the initial sort, sorting continues as new data is added or inserted. By sorting IP addresses in order, organizations can significantly accelerate IP address lookups within a particular range. Yellowbrick also uses zone mapping to tell the database which blocks of the NVMe flash contain which ranges of data.

In the example in Figure 1 below, the IP addresses are sorted into a column. Queries for an address in the 16.0.0.0/1 range only require a scan 1/14th of the data.  In a real-world use case, where the dataset is larger and the query is smaller, this approach can deliver a 99% increase in read efficiency.  Yellowbrick even displays read efficiency to users to help with data layout and query planning.

Figure 1. Zone Map

A real-world example: massive scalability

To illustrate how well Yellowbrick scales in the face of massive data growth, consider the following example.

We have two tables, one table called netflow and another called netflow_ip.  Netflow has the IP address stored as a varchar, like most databases, while netflow_ip uses the Yellowbrick “IP address” data type. The dataset we are using is an open source dataset that has real metrics but the IP addresses have been randomized.

The DLL for the table with the Yellowbrick IP address data type is as follows:

Notice that this table clusters on timestamp, source and destination IP.  This sorts the table on all three columns, enabling us to get maximum read efficiency if we search by source or destination IP and data ranges.

A Simple IP lookup test

A simple query that counts all the records where an IP address matches with the varchar implementation looks like the following:

It takes 4.2 seconds to scan all the rows (14.38 Billion) and count them.

But if we use the Yellowbrick IP address data type, and sort or cluster the data like the DDL above, we see a slash the response time from 4.2 seconds down to 203ms (less than a quarter of a second).  A 20x improvement thanks to native flash queries and zone maps!

But how does this implementation scale as you add data? To test scalability, we can start with a table with the same DDL as described above. We then load one week at a time and run the same query each time as we load data.

Our example query looks for IP addresses that accessed the 64.124.9.0/24 range over the entire dataset. Here is the query:

At week one, we have 500 million records. The number grows to the full 14.28 billion by week 21.  Response times start at 58.644 milliseconds or .0058 seconds against ~500 million rows and by the time we’ve increased the data size 28x to 14.28 billion, the query response time is 122 milliseconds or .0122 seconds. In short, increasing the number of records queried from 500 million to 14 billion merely doubled the response time, and response times are still under a quarter of a second. Figure 2 plots the response times against the data growth, in seconds.

Figure 2. Yellowbrick completes queries over 28x the size in just double the time (.0122 seconds).

Load data at 10-million rows per second

We’ve shown how Yellowbrick deliver high query performance and scales that performance with data growth. But another area where enterprises struggle with cybersecurity is loading data into the system to begin with. With Yellowbrick, organizations can perform SQL inserts over ODBC, JDBC, and ADO.net.  As many are aware, while these protocols are known to be simple and easy to use, they don’t generally move big data sets very quickly. In the case of Yellowbrick, organizations can load about ~150,000 rows a second using these connectors. To enable organizations to load data more rapidly, Yellowbrick created a versatile and high performance data loading tool called ybload. In the next example I will use bare-metal server equipped with SSDs as a source and load data directly into Yellowbrick using ybload  10G networking.

The following test loads about a month’s worth of Netflow data, which amounts to about 308GB.

77G -rw-rw-r--. 1 cshrauder cshrauder  77G Jun 27 2018 august.week1.csv
76G -rw-rw-r--. 1 cshrauder cshrauder  76G Apr 3 2018 august.week2.csv
74G -rw-rw-r--. 1 cshrauder cshrauder  74G Apr 3 2018 august.week3.csv
78G -rw-rw-r--. 1 cshrauder cshrauder  78G Apr 3 2018 august.week4.csv
3.7G -rw-rw-r--. 1 cshrauder cshrauder 3.7G Apr  2 2018 august.week5.csv

I can load the data with a simple ybload command:

ybload -h  --username  -d netflow -t netflow ./august.week* 
--nullmarker " "   --bad-row-file /home/cshrauder/log/august.bad --logfile 
/home/cshrauder/log/august.log --read-sources-concurrently always  
--num-readers 16 --num-parsers-per-reader 4

The data set is loaded at over 10 million rows per second, at about 1GB/s.

At these speeds the entire 308GB dataset is loaded and ready for query in 5 minutes and 15 seconds.

Conclusion

Yellowbrick offers a simple solution to the massive scale and latency requirements associated with production level Netflow analysis. By utilizing the latest hardware and software approaches, Yellowbrick delivers a solution that can ingest Netflow and other data sets at line rate while still serving queries and other application needs.  Yellowbrick can make the data available immediately and there are no indexes to create and manage. Data sets over 2 petabytes are easily ingested and managed with a single Yellowbrick cluster occupying only 10U of rack space. If you find yourself questioning any of these performance or scale numbers, seeing is believing. The Yellowbrick team will be happy to set you up with a proof of concept system so you can try it for yourself.

Flashback Recovery in MariaDB/MySQL/Percona

$
0
0

Feed: Planet MySQL
;
Author: MyDBOPS
;

In this blog, we will see how to do flashback recovery or rolling back the data in MariaDB, MySQL and Percona.

As we know the saying  “All human make mistakes”, following that in Database environment the data can be deleted or updated in the database either by intentionally or by accidentally.

To recover the lost data we have multiple ways.

  • The data can be recovered from the latest full backup or incremental backup when data size is huge it could take hours to restore it.
  • From backup of Binlogs.
  • Data can also be recovered from delayed slaves, this case would be helpful when the mistake is found immediately, within the period of delay.

The above either way can help to recover the lost data, but what really matters is, What is the time taken to rollback or recover the data? and How much downtime was taken to get back to the initial state?

To overcome this disaster mysqlbinlog has a very useful option i.e –flashback that comes along with binary of MariaDB server though it comes with Mariaserver, it works well with Oracle Mysql servers and Percona flavour of MySQL.

What is Flashback?

Restoring back the data to the previous snapshot in a MySQL database or in a table is called Flashback.

Flashback options help us to undo the executed row changes(DML events).

For instance, it can change DELETE events to INSERTs and vice versa, and also it will swap WHERE and SET parts of the UPDATE events.

Prerequisites for using flashback :

  • binlog_format = ROW
  • binlog_row_image = FULL

Let us simulate a few test cases where flashback comes as a boon for recovering data.

For simulating the test cases I am using employees table and mariadb version 10.2

MariaDB [employees]> select @@version;
+---------------------+
| @@version           |
+---------------------+
| 10.2.23-MariaDB-log |
+---------------------+
1 row in set (0.02 sec)

Table structure :

MariaDB [employees]> show create table employeesG
*************************** 1. row ***************************
       Table: employees
Create Table: CREATE TABLE `employees` (
  `emp_no` int(11) NOT NULL,
  `birth_date` date NOT NULL,
  `first_name` varchar(14) NOT NULL,
  `last_name` varchar(16) NOT NULL,
  `gender` enum('M','F') NOT NULL,
  `hire_date` date NOT NULL,
  PRIMARY KEY (`emp_no`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

Case 1:  Rollbacking the Deleted data.

Consider the data is deleted was from employees table where first_name =’Chirstian’ .

MariaDB [employees]> select COUNT(*) from employees where first_name ='Chirstian';
+----------+
| COUNT(*) |
+----------+
|      226 |
+----------+
1 row in set (0.07 sec)

MariaDB [employees]> delete from employees where first_name ='Chirstian';
Query OK, 226 rows affected (0.15 sec)

To revert the data to the intial state ,we need to decode the binlog and get the start and stop position of the delete event happened to the employees table.

It is necessary to take a proper start and stop positions. Start position should be taken exactly after BEGIN and Stop position is before the final COMMIT.

[root@vm3 vagrant]# mysqlbinlog -v --base64-output=DECODE-ROWS /var/lib/mysql/mysql-bin.000007 > mysql-bin.000007.txt
BEGIN
/*!*/;
# at 427
# at 501
#190417 17:49:49 server id 1  end_log_pos 501 CRC32 0xc7f1c84b  Annotate_rows:
#Q> delete from employees where first_name ='Chirstian'
#190417 17:49:49 server id 1  end_log_pos 569 CRC32 0x6b1b5c98  Table_map: `employees`.`employees` mapped to number 29
# at 569
#190417 17:49:49 server id 1  end_log_pos 7401 CRC32 0x6795a972         Delete_rows: table id 29 flags: STMT_END_F
### DELETE FROM `employees`.`employees`
### WHERE
###   @1=10004
###   @2='1954:05:01'
###   @3='Chirstian'
###   @4='Koblick'
###   @5=1
###   @6='1986:12:01'
# at 23733
#190417 17:49:49 server id 1  end_log_pos 23764 CRC32 0xf9ed5c3e        Xid = 455
### DELETE FROM `employees`.`employees`
### WHERE
### @1=498513
### @2='1964:10:01'
### @3='Chirstian'
### @4='Mahmud'
### @5=1
### @6='1992:06:03'
# at 7401
COMMIT/*!*/;
# at 23764
#190417 17:49:49 server id 1  end_log_pos 23811 CRC32 0x60dfac86        Rotate to mysql-bin.000008  pos: 4
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;

Once the count is verified the from the taken positions, we can prepare the data file or the .sql file using flashback as below

[root@vm3 vagrant]# mysqlbinlog  -v --flashback --start-position=427 --stop-position=7401 /var/lib/mysql/mysql-bin.000007  > insert.sql

Below is the comparison of conversion from Delete to Insert for a single record:

### DELETE FROM `employees`.`employees`
### WHERE
### @1=498513
### @2='1964:10:01'
### @3='Chirstian'
### @4='Mahmud'
### @5=1
### @6='1992:06:03'

### INSERT INTO `employees`.`employees`
### SET
### @1=498513
### @2='1964:10:01'
### @3='Chirstian'
### @4='Mahmud'
### @5=1
### @6='1992:06:03'
MariaDB [employees]> source insert.sql
Query OK, 0 rows affected (0.01 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.01 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)

And the count is verified after the data load.

MariaDB [employees]> select COUNT(*) from employees where first_name ='Chirstian';
+----------+
| COUNT(*) |
+----------+
|      226 |
+----------+
1 row in set (0.06 sec)

Case 2 :  Rollbacking the Updated data.

The data was updated based on below conditions

MariaDB [employees]> select COUNT(*) from employees where first_name ='Chirstian' and gender='M';
+----------+
| COUNT(*) |
+----------+
|      129 |
+----------+
1 row in set (0.14 sec)

MariaDB [employees]> update employees set gender='F' where first_name ='Chirstian' and gender='M';
Query OK, 129 rows affected (0.16 sec)
Rows matched: 129  Changed: 129  Warnings: 0

MariaDB [employees]> select COUNT(*) from employees where first_name ='Chirstian' and gender='M';
+----------+
| COUNT(*) |
+----------+
|        0 |
+----------+
1 row in set (0.07 sec)

To revert back the updated data, the same steps to be followed as in case 1.

[root@vm3 vagrant]# mysqlbinlog -v --flashback --start-position=427 --stop-position=8380 /var/lib/mysql/mysql-bin.000008 > update.sql

MariaDB [employees]> source update.sql
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.00 sec)
Query OK, 0 rows affected (0.01 sec)
Query OK, 0 rows affected (0.00 sec)

MariaDB [employees]> select COUNT(*) from employees where first_name ='Chirstian' and gender='M';
+----------+
| COUNT(*) |
+----------+
|      129 |
+----------+
1 row in set (0.06 sec)

In the above two cases by using flashback option we were able to change Event Type statements from DELETE to INSERT and Update_Event statements by Swapping the SET part and WHERE part.

There are few Limitations of Flashback 

  • It Doesn’t support DDL ( DROP/TRUNCATE or other DDL’s)
  • It Doesn’t support encrypted binlog
  • It Doesn’t support compressed binlog

Key Takeaways:

  • To reverse the mishandled operations from binary logs.
  • No need to stop the server to carry out this operation.
  • When the data is small to revert back, flashback process is very faster than recovering the data from Full Backup.
  • Point in time recovery (PITR) becomes easy.

Photo by Jiyeon Park on Unsplash

Advertisements

Fun with Bugs #85 – On MySQL Bug Reports I am Subscribed to, Part XX

$
0
0

Feed: Planet MySQL
;
Author: Valeriy Kravchuk
;

We have a public holiday here today and it’s raining outside for a third day in a row already, so I hardly have anything better to do than writing yet another review of public MySQL bug reports that I’ve subscribed to recently.

Not sure if these reviews are really considered useful by anyone but few of my readers, but I am still going to try in a hope to end up with some useful conclusions. Last time I’ve stopped on Bug #94903, so let me continue with the next bug in my list:

  • Bug #94912 – “O_DIRECT_NO_FSYNC possible write hole“. In this bug report Janet Campbell shared some concerns related to the way O_DIRECT_NO_FSYNC (and O_DIRECT) settings for innodb_flush_method work. Check comments, including those by Sunny Bains, where he agrees that “…this will cause problems where the redo and data are on separate devices.”. Useful reading for anyone interested in InnoDB internals or using  innodb_dedicated_server setting in MySQL 8.0.14+.
  • Bug #94971 – “Incorrect key file error during log apply table stage in online DDL“. Monty Solomon reported yet another case when “online’ ALTER for InnoDB table fails in a weird way. The bug is still “Open” and there is no clear test case to just copy/paste, but both the problem and potential solutions (make sure you have “big enough” innodb_online_alter_log_max_size or better use pt-online-schema-change or gh-ost tools) were already discussed here.
  • Bug #94973 – “Wrong result with subquery in where clause and order by“. Yet another wrong results bug with subquery on MySQL 5.7.25 was reported by Andreas Kohlbecker. We can only guess if MySQL 8 is also affected (MariaDB 10.3.7 is not, based on my test results shared below) as Oracle engineer who verified the bug had NOT card to check or share the results of this check. What can be easier than running this (a bit modified) test case on every MySQL major version and copy pasting the results:

    MariaDB [test]> CREATE TABLE `ReferenceB` (
        ->   `id` int(11) NOT NULL,
        ->   `bitField` bit(1) NOT NULL,
        ->   `refType` varchar(255) NOT NULL,
        ->   `externalLink` longtext,
        ->   PRIMARY KEY (`id`)
        -> ) ENGINE=MyISAM DEFAULT CHARSET=utf8;
    Query OK, 0 rows affected (0.170 sec)

    MariaDB [test]> INSERT INTO ReferenceB (id, bitField, refType, externalLink) VALUES(1, 0, ‘JOU’, NULL);
    Query OK, 1 row affected (0.027 sec)

    MariaDB [test]> INSERT INTO ReferenceB (id, bitField, refType, externalLink) VALUES(2, 0, ‘JOU’, NULL);
    Query OK, 1 row affected (0.002 sec)

    MariaDB [test]> SELECT hex(bitField) from ReferenceB  where id in (select id as
    y0_ from ReferenceB  where refType=’JOU’) order by externalLink asc;
    +—————+
    | hex(bitField) |
    +—————+
    | 0             |
    | 0             |
    +—————+
    2 rows in set (0.028 sec)

    But we do not see anything like that in the bug report… This is sad.

  • Bug #94994 – “Memory leak detect on temptable storage engine“. Yet another memory leak (found with ASan) reported by Zhao Jianwei, who had also suggested a patch.
  • Bug #95008 – “applying binary log doesn’t work with blackhole engine tables“. This bug was reported by Thomas Benkert. It seems there is a problem to apply row-based events to BLACKHOLE table and this prevents some nice recovery tricks from working.
  • Bug #95020 – “select no rows return but check profile process Creating sort index“. Interesting finding from cui jacky. I can reproduce this with MariaDB as well. It seems we either have to define some new stage or define “Creating sort index” better than in the current manual. This:

    The thread is processing a
    SELECT that is resolved using
    an internal temporary table.

    is plain wrong in the case shown in the bug report IMHO.

  • Bug #95040 – “Duplicately remove locks from lock_sys->prdt_page_hash in btr_compress“. One of those rare cases when Zhai Weixiang does not provide the patch, just suggests the fix based on code review 🙂
  • Bug #95045 – “Data Truncation error occurred on a write of column 0Data was 0 bytes long and“. This really weird regression bug in MySQL 8.0.14+ was reported by Adarshdeep Cheema. MariaDB 10.3 is surely not affected.
  • Bug #95049 – “Modified rows are not locked after rolling back to savepoint“. Bug reporter, John Lin, found that fine MySQL manual does not describe the real current implementation. Surprise!
  • Bug #95058 – “Index not used for column with IS TRUE or IS FALSE operators“. Take extra care when using BOOLEAN columns in MySQL. As it was noted by Monty Solomon, proper index is NOT used when you try to check BOOLEAN values as manual suggests, using IS TRUE or IS FALSE conditions. Roy Lyseng explained how such queries are threated internally, but surely there is a better way. MariaDB 10.3.7 is also affected, unfortunately.
  • Bug #95064 – “slave server may has gaps in Executed_Gtid_Set when a special case happen “. Nice bug report from yoga yoga, who had also contributed a patch. Parallel slave can easily get out of sync with master in case of lock wait timeout and failed retries. Again, we do NOT see any check if MySQL 8 is affected, unfortunately.
  • Bug #95065 – “Strange memory management when using full-text indexes“. We all know that InnoDB FULLTEXT indexes implementation is far from perfect. Now, thanks to Yura Sorokin, we know also about a verified memory leak bug there that may lead to OOM killing of MySQL server.
  • Bug #95070 – “INSERT .. VALUES ( .., (SELECT ..), ..) takes shared lock with READ-COMMITTED“. Seunguck Lee found yet another case of InnoDB locking behavior that MySQL manual does not explain. The bug is still “Open” for some reason.
  • Bug #95115 – “mysqld deadlock of all client threads originating from 3-way deadlock“. It took some efforts for bug reporter, Sandeep Dube, and other community users (mostly Jacek Cencek) to attract proper attention to this bug from proper Oracle developer, Dmitry Lenev, until it ended up “Verified” based on code review. We still can not be sure if MySQL 8 is also affected.

That’s all for now. I have few more new bug reports that I monitor, but I do not plan to continue with this kind of reviews in upcoming few months in this blog. I hope I’ll get a reason soon to write different kind of posts, with more in depth study of various topics…

In any case you may follow me on Twitter for anything related to recent interesting or wrongly handled MySQL bug reports.

This view of Chelsea from our apartment at Chelsea Cloisters reminds me that last year I spent spring holiday season properly – no time was devoted to MySQL bugs 🙂

To summarize:

  1. Do not use O_DIRECT_NO_FSYNC value for innodb_flush_method if your redo logs are located on different device than your data files. Just don’t.
  2. Some Oracle engineers who process bugs still do not care to check if all supported major versions are affected and/or share the results of such checks in public.
  3. There are still many details of InnoDB locking to study, document properly and maybe fix.
  4. I am really concerned with the state of MySQL optimizer. We see all kinds of weird bugs (including regressions) and very few fixes in each maintenance release.

How to create multiple accounts for an app?

$
0
0

Feed: Planet MySQL
;
Author: MySQL Server Dev Team
;

This is a 3 part blog series:

You can now grant CREATE USER so that your web apps would be able to use multiple accounts without you risking the end user hijacking the database by changing your root credentials for example.  Wouldn’t it be nice, if you could grant a user privileges to create or modify users except a few users? If you have such use cases, then this blog post will interest you.

First, let us understand how you can modify a user’s specifications. There are following two ways to do that.

  1. Through a DDL statements if you have ‘CREATE USER’ privilege.

For instance:

Now, foo has the ability to do the following:

  1. Through a DML statements if you have DB-level privileges on the mysql database.

For instance:

Now, foo has the ability to do the following:

In both cases, foo was able to modify the root account. What if you  want to protect accounts such as root from being modified by other users?

In MySQL 8.0.16, we have added the following two capabilities.

  • A SYSTEM_USER privilege. It prevents users who have that privilege from being modified by the users who have privileges to create or modify users, but do not have the SYSTEM_USER privilege.  You may read more about SYSTEM_USER in this blog post.
  • Revoke privileges from database objects, even if the user is granted global privileges. You may read more about privilege restrictions in this blog post.

This is the final blog post in the series that shows how to protect users leveraging the preceding two capabilities.  Let us understand doing that through the following example.

The discussion here assumes that ‘partial_revokes’ system variable is ON.

We create two users and grant the CREATE USER privilege to them.

Grant the SYSTEM_USER privilege to one user.

Grant the global update privilege to other user, but revoke privileges on the mysql database.

bar_admin cannot change foo_admin using DDL statements because foo_admin has SYSTEM_USER but bar_admin does not.

bar_admin cannot change foo_admin because privileges on the mysql database have been revoked.

Thus, you created a user ‘foo_admin’ who cannot be modified by another user even though the latter has privileges to modify some users. For bar_admin to modify foo_admin, bar_admin must also have the SYSTEM_USER privilege.

Let us create another user who has ‘CREATE USER’ and ‘SYSTEM_USER’ privileges. This user can modify the properties of user ‘foo_admin’.

baz_admin can change the password of foo_admin.

As we saw, in order to modify the properties of a user who is granted the SYSTEM_USER privilege, you need to have CREATE USER as well as SYSTEM_USER privilege.

Based on the preceding observations, we may visualize users with respect to the SYSTEM_USER and CREATE USER privileges as following.

System Users:  

Users who are granted at least ‘SYSTEM_USER’ privilege, but not the CREATE USER privilege. These users themselves have no capability to modify other users. These users can only be modified by power users.

Privileged Users

The users who are granted at least ‘CREATE USER’ privilege, but not the ‘SYSTEM_USER’ privilege. These users can modify the all users except system users.

Non-privileged Users

The users who have neither ‘SYSTEM_USER’ nor ‘CREATE USER’ privilege, but may be granted other privileges. These users cannot modify any other users.

Power Users:

The users who are granted at least the ‘SYSTEM_USER’ and ‘CREATE USER’ privileges.  These users can modify any user available in the database. These users are most powerful users hence named as power users.

The strategy to create the immutable users

  • Evaluate carefully which users really need to be granted the SYSTEM_USER privilege. There should not be many users who will need the SYSTEM_USER privilege.
  • To protect users against being modified through DDL statements, grant them the SYSTEM_USER privilege. This prevents them from being modified by users who do not have SYSTEM_USER. Your root account will definitely be one of them. Not surprisingly, it is granted the SYSTEM_USER privilege by default.
  • To protect against being modified through DML statements, impose a partial revoke on the mysql database on administrative users. First, you may create an administrative user who has DB-related privileges, granted globally but revoked for mysql database.  You may use this administrative user to grant privileges to other users.  You could achieve the same effect through roles and making them default as well. Whatever suits you best?

Let us see how to do that through a role.

Create an administrator user who needs the global access of DB-level privileges.  We can achieve this by granting the previously created role to the user.

Connect as the administrator and activate the role to get the required privileges.

Conclusion

In this blog post we learned to create users who are can be protected from getting modified by the users who usually create and modify users.

We hope you found this blog post series informative and interesting.  Please give the techniques described here a try and let us know your feedback.

Thank you for using MySQL




Viewing all 275 articles
Browse latest View live