Feed: Cloud – Cloudera Engineering Blog.
Author: Shelby Khan.

Fig 1 – Architecture

Introduction

Many of Cloudera’s customers set up Cloudera Manager to collect their clusters’ diagnostic data on a regular schedule and automatically send that data to Cloudera. Cloudera analyzes this data, identifies potential problems in the customer’s environment, and alerts customers, requiring fewer back-and-forths with our customers when they file a support case and provides Cloudera with critical information to improve future versions of all of Cloudera’s software. If Cloudera discovers a serious issue, Cloudera searches this diagnostic data and proactively notifies Cloudera customers who might encounter problems due to the issue. This blog post explains how Cloudera internally uses the Altus as a Service platform in the cloud to perform these analyses. Offloading processing and ad-hoc visualization to the cloud reduces costs since compute resources are used only when needed. Transient ETL workloads process incoming data and stateless data warehouse clusters are used for ad-hoc visualizations. The clusters share metadata via Altus SDX.

Overview

In Cloudera EDH, diagnostic bundles are ingested from customer environments and normalized using Apache NiFi. The bundles are stored in an AWS S3 bucket with date-based partitions (Step 1 in Fig 1). AWS Lambda is scheduled daily (Step 2 in Fig 1) to process the previous day’s data (Step 3 in Fig 1) by spinning up Altus Data Engineering clusters to execute ETL processing and terminating the clusters on job completion. The jobs on these DE clusters produce a fact table and three dimension tables (Star schema). This extracted data is stored in a different S3 bucket (Step 4 in Fig 1) and the metadata produced from these processes, such as schema and partitions, are stored in Altus SDX (Step 5 in Fig 1). Whenever data needs to be visualized, a stateless Altus Data Warehouse cluster is created, which provides easy access to the data via the built-in SQL Editor or JDBC/ODBC Impala connector (Step 6,7 in Fig 1).

Altus SDX

We create configured SDX namespaces for sharing metadata and fine-grained authorization between workloads that run on clusters that we create in Altus Data Engineering and Data Warehouse. Configured SDX namespaces are built on Hive metastore and Sentry databases. These databases are set up and managed by Altus customers.

First, we will create two databases in an external database.

Fig 2 – Hive and Sentry external database creation

We will then create a configured namespace using those databases.

Fig 3 – Configured namespace creation

Altus will initialize the schemas of Hive metastore and Sentry databases when a cluster using a configured namespace is started. We chose to grant namespace admin group all privileges so that when the internal user ‘altus’, who executes the Spark job in Data Engineering clusters, has necessary DDL permissions.

Altus Data Engineering

We need to process the bundles that were uploaded the previous day. To do this, we need a mechanism to start an Altus Data Engineering cluster periodically, which executes the job and terminates after the processing is done. We execute an AWS Lambda function that is periodically triggered by an AWS Cloudwatch rule. This AWS Lambda uses Altus Java SDK to kick off an ephemeral Data Engineering cluster with a Spark job.

This will process the diagnostic data that was stored in a S3 bucket. It will create a fact table and three dimension tables. The data for those tables will be stored in a different S3 bucket. The metadata will be stored in Altus SDX. The cluster is deleted at the end as the job only runs for 30 minutes.

Altus Data Warehouse

Whenever we want to analyze the data, we spin up an Altus Data Warehouse cluster in the same environment, using the same namespace as the Altus Data Engineering cluster. Doing so allows the Altus Data Warehouse to have direct access to the data and metadata created by the Altus Data Engineering clusters.

Fig 4 – Data Warehouse cluster creation

Once the cluster is created we can use the built-in Query Editor to quickly analyze the data.

Fig 5 – Query Data Warehouse cluster using SQL Editor

If you need to generate custom dashboards, connect  to the Altus Data Warehouse clusters using Cloudera Impala JDBC/ODBC connector. Recently, we released a new Impala JDBC driver which can connect to an Altus Data Warehouse cluster by simply specifying the cluster’s name.

Fig 6 – Data Warehouse JDBC connection using cluster name

In the image above, we show how easy it is to connect Tableau to the tables in Altus Data Warehouse.

Fig 7 – Visualizing distribution of services in the Cloudera clusters

In the above visualization, you can see the distribution of services running in Cloudera clusters. Note that this is based on a random dataset.

What’s Next?

To get started with a 30-day free Altus trial, visit us at https://www.cloudera.com/products/altus.html. Send questions or feedback to the Altus community forum.

Feed: AWS Big Data Blog.

Amazon Athena is a serverless query engine for data on Amazon S3. Many customers use Athena to query application and service logs, schedule automated reports, and integrate with their applications, enabling new analytics-based capabilities.

Different types of users rely on Athena, including business analysts, data scientists, security, and operations engineers. But how do you separate and manage these workloads so that users get the best experience while minimizing costs?

In this post, I show you how to use workgroups to do the following:

• Separate workloads.
• Control user access.
• Manage query usage and costs.

Separate workloads

By default, all Athena queries execute in the primary workgroup.  As an administrator, you can create new workgroups to separate different types of workloads.  Administrators commonly turn to workgroups to separate analysts running ad hoc queries from automated reports.  Here’s how to build out that separation.

First create two workgroups, one for ad hoc users (ad-hoc-users) and another for automated reports (reporting).

Next, select a specific output location. All queries executed inside this workgroup save their results to this output location. Routing results to a single secure location helps make sure users only access data they are permitted to see. You can also enforce encryption of query results in S3 by selecting the appropriate encryption configuration.

Workgroups also help you simplify the onboarding of new users to Athena. By selecting override client-side settings, you enforce a predefined configuration on all queries within a workgroup. Users no longer have to configure a query results output location or S3 encryption keys. These settings default to the parameters defined for the workgroup where those queries execute. Additionally, each workgroup maintains a unique query history and saved query inventory, making queries easier for you to track down.

Finally, when creating a workgroup, you can add up to 50 key-value pair tags to help identify your workgroup resources. Tags are also useful when attempting to allocate Athena costs between groups of users. Create Name and Dept tags for the ad-hoc-users and reporting workgroups with their name and department association.

Control user access to workgroups

Now that you have two workgroups defined, ad-hoc-users and reporting, you must control who can use and update them.  Remember that workgroups are IAM resources and therefore have an ARN. You can use this ARN in the IAM policy that you associate with your users.  In this example, create a single IAM user representing the team of ad hoc users and add the individual to an IAM group. The group contains a policy that enforces what actions these users can perform.

Start by reviewing IAM Policies for Accessing Workgroups and Workgroup Example Policies to familiarize yourself with policy options. Use the following IAM policy to set up permissions for your analyst user. Grant this user only the permissions required for working in the ad-hoc-users workgroup. Make sure that you tweak this policy to match your exact needs:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "athena:ListWorkGroups"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "athena:StartQueryExecution",
                "athena:GetQueryResults",
                "athena:DeleteNamedQuery",
                "athena:GetNamedQuery",
                "athena:ListQueryExecutions",
                "athena:StopQueryExecution",
                "athena:GetQueryResultsStream",
                "athena:GetQueryExecutions",
                "athena:ListNamedQueries",
                "athena:CreateNamedQuery",
                "athena:GetQueryExecution",
                "athena:BatchGetNamedQuery",
                "athena:BatchGetQueryExecution",
                "athena:GetWorkGroup",
                "athena:ListTagsForResource"
            ],
            "Resource": "arn:aws:athena:us-east-1:112233445566:workgroup/ad-hoc-users"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": "arn:aws:s3:::demo/workgroups/adhocusers/*"
        },
{
            "Effect": "Allow",
            "Action": [
                "glue:Get*"
            ],
            "Resource": [
                "arn:aws:glue:us-east-1:112233445566:catalog",
                "arn:aws:glue:us-east-1:112233445566:database/amazon",
                "arn:aws:glue:us-east-1:112233445566:table/amazon/*"
            ]
        }
    ]
}

Now your analyst user can execute queries only in the ad-hoc-users workgroup. The analyst user can switch to other workgroups, but they lose access when they try to perform any action. They are further restricted to list and query only those tables that belong to the Amazon database. For more information about controlling access to AWS Glue resources such as databases and tables, see AWS Glue Resource Policies for Access Control.

The following screenshot shows what the analyst user sees in the Athena console:

I’ve created a simple Node.js tool that executes SQL queries stored as files in a given directory. You can find my Athena test runner code in the athena_test_runner GitHub repo. You can use this code to simulate a reporting tool, after configuring it to use a workgroup. To do that, create an IAM role with permissions like those previously defined for the analyst user. This time, restrict access to the reporting workgroup.

The following JavaScript code example shows how to select a workgroup programmatically when executing queries:

function executeQueries(files) {
    params = 
    {
      "QueryString": "", 
      "ResultConfiguration": { 
        "OutputLocation": ""
      },
      "QueryExecutionContext": {
        "Database": "default"
      },
      "WorkGroup":"reporting"
    }
 
    params.QueryString = "SELECT * FROM amazon.final_parquet LIMIT 10"
    return new Promise((resolve, reject) => {
        athena.startQueryExecution(params, (err, results) => {
            if (err) {
                reject(err.message)
            } else {
                resolve(results)
            }
        })
    })
}

Run sample automated reports under the reporting workgroup, with the following command:

node index.js testsuite/

Query histories remain isolated between workgroups. A user logging into the Athena console as an analyst using the ad-hoc-users workgroup doesn’t see any automated reports that you ran under the reporting workgroup.

Managing query usage and cost

You have two workgroups configured: one for ad hoc users and another for automated reports. Now, you must safeguard against bad queries. In this use case, two potential situations for query usage should be monitored and controlled:

• Make sure that users don’t run queries that scan more data than allowed by their budget.
• Safeguard against automated script bugs that could cause indefinite query retirement.

First, configure data usage controls for your ad-hoc-users workgroup. There are two types of data usage controls: per-query and per-workgroup.

Set the per-query control for analysts to be 1 GB. This control cancels any query run in the ad-hoc-users workgroup that tries to scan more than 1 GB.

To observe this limit in action, choose Update, return to the query editor, and run a query that would scan more than 1 GB. This query triggers the error message, “Query cancelled! : Bytes scanned limit was exceeded”. Remember that you incur charges for data the query scanned up to the point of cancellation. In this case, you incur charges for 1 GB of data.

Now, switch to your reporting workgroup. For this workload, you’re not worried about individual queries scanning too much data. However, you want to control the aggregate amount of data scanned of all queries in this workgroup.

Create a per-workload data usage control for the reporting workgroup. You can configure the maximum amount of data scanned by all queries in the workgroup during a specific period.

For the automated reporting workload, you probably have a good idea of how long the process should take and the total amount of data that queries scan during this time. You only have a few reports to run, so you can expect them to run in a few minutes, only scanning a few megabytes of data. Begin by setting up a low watermark alarm to notify you when your queries have scanned more data than you would expect in five minutes. The following example is for demo purposes only. In most cases, this period would be longer. I configured the alarm to send a notification to an Amazon SNS topic that I created.

To validate the alarm, I made a minor change to my test queries, causing them to scan more data. This change triggered the SNS alarm, shown in the following Amazon CloudWatch dashboard:

Next, create a high watermark alarm that is triggered when the queries in your reporting workgroup exceed 1 GB of data over 15 minutes. In this case, the alarm triggers an AWS Lambda function that disables the workgroup, making sure that no additional queries execute in it. This alarm protects you from incurring faulty automation code or runaway query costs.

Before creating the data usage control, create a Node.js Lambda function to disable the workgroup. Paste in the following code:

exports.handler = async (event) => {
    const AWS = require('aws-sdk')
    let athena = new AWS.Athena({region: 'us-east-1'})
 
    let msg = JSON.parse(event.Records[0].Sns.Message)
    let wgname = msg.Trigger.Dimensions.filter((i)=>i.name=='WorkGroup')[0].value
    
    athena.updateWorkGroup({WorkGroup: wgname, State: 'DISABLED'})
 
    const response = {
        statusCode: 200,
        body: JSON.stringify(`Workgroup ${wgname} has been disabled`),
    };
    return response;
}

This code grabs the workgroup name from the SNS message body and calls the UpdateWorkGroup API action with the name and the state of DISABLED. The Athena API requires the most recent version of the AWS SDK. When you create the Lambda bundle, include the latest AWS SDK version in that bundle.

Next, create a new SNS topic and a subscription. For Protocol, select AWS Lambda. Then, select the Lambda function that you created in the previous step.

In the Athena console, create the second alarm, 1 GB for 15 min., and point it to the SNS topic that you created earlier. When triggered, this SNS topic calls the Lambda function that disables the reporting workgroup. No more queries can execute in this workgroup. You see this error message in the console when a workgroup is disabled:

Athena exposes other aggregated metrics per workgroup under the AWS/Athena namespace in CloudWatch, such as the query status and the query type (DDL or DML) per workgroup. To learn more, see Monitoring Athena Queries with CloudWatch Metrics.

Cost allocation tags

When you created your ad-hoc-users and reporting workgroups, you added Name and Dept tags. These tags can be used in your Billing and Cost Management console to determine the usage per workgroup.

Summary

In this post, you learned how to use workgroups in Athena to isolate different query workloads, manage access, and define data usage controls to protect yourself from runaway queries. Metrics exposed to CloudWatch help you monitor query performance and make sure that your users are getting the best experience possible. For more details, see Using Workgroups to Control Query Access.

About the Author

Roy Hasson is a Global Business Development Manager for AWS Analytics. He works with customers around the globe to design solutions to meet their data processing, analytics and business intelligence needs. Roy is big Manchester United fan, cheering his team on and hanging out with his family.

Feed: Planet PostgreSQL.

2018–01–12 16:54:40.208 MSK pid=20949,user=user_3,db=test,host=127.0.0.1:55763 LOG: process 20949 still waiting for AccessShareLock on relation 10000 of database 9000 after 5000.055 ms2018–01–12 16:54:40.208 MSK pid=20949,user=user_3,db=test,host=127.0.0.1:55763 DETAIL: Process holding the lock: 46091. Wait queue: 18639, 20949, 53445, 20770, 10799, 47217, 37659, 6727, 37662, 25742, 20771,

if masterthen falseif lag > maxthen create file and return falseif lag > min and file existsthen return falseif lag < min and file existsthen remove file and return trueelsetrue

Performing Upgrade------------------Analyzing all rows in the new cluster okFreezing all rows on the new cluster okDeleting files from new pg_clog okCopying old pg_clog to new server okSetting next transaction ID and epoch for new cluster okDeleting files from new pg_multixact/offsets okCopying old pg_multixact/offsets to new server okDeleting files from new pg_multixact/members okCopying old pg_multixact/members to new server okSetting next multixact ID and offset for new cluster okResetting WAL archives okconnection to database failed: FATAL: database "template1" does not existcould not connect to new postmaster started with the command:"/home/test/inst/pg9.6/bin/pg_ctl" -w -l "pg_upgrade_server.log" -D "new/"-o "-p 50432 -b -c synchronous_commit=off -c fsync=off -cfull_page_writes=off -c listen_addresses='' -c unix_socket_permissions=0700-c unix_socket_directories='/home/test/tmp/u'" startFailure, exitingTest script:#!/bin/bashPGOLD=~/inst/pg9.4/binPGNEW=~/inst/pg9.6/bin${PGOLD}/pg_ctl init -s -D old -o "--lc-messages=C -T pg_catalog.english"${PGNEW}/pg_ctl init -s -D new -o "--lc-messages=C -T pg_catalog.english"echo vacuum_defer_cleanup_age=10000 >> new/postgresql.auto.conf# move txid to 3 000 000 000 in old cluster as in production${PGOLD}/pg_ctl start -w -D old -o "--port=54321--unix_socket_directories=/tmp"${PGOLD}/vacuumdb -h /tmp -p 54321 --all --analyze${PGOLD}/vacuumdb -h /tmp -p 54321 --all --freeze${PGOLD}/pg_ctl stop -D old -m smart#${PGOLD}/pg_resetxlog -x 3000000000 olddd if=/dev/zero of=old/pg_clog/0B2D bs=262144 count=1# # move txid in new cluster bigger than vacuum_defer_cleanup_age may fixproblem# ${PGNEW}/pg_ctl start -w -D new -o "--port=54321--unix_socket_directories=/tmp"# echo "select txid_current();" | ${PGNEW}/pgbench -h /tmp -p 54321 -n -P 5-t 100000 -f- postgres# ${PGNEW}/pg_ctl stop -D new -m smart${PGNEW}/pg_upgrade -k -d old/ -D new/ -b ${PGOLD}/ -B ${PGNEW}/# rm -r new old pg_upgrade_* analyze_new_cluster.sh delete_old_cluster.sh

Feed: AWS Big Data Blog.

Amazon Athena is a serverless query engine for data on Amazon S3. Many customers use Athena to query application and service logs, schedule automated reports, and integrate with their applications, enabling new analytics-based capabilities.

Different types of users rely on Athena, including business analysts, data scientists, security, and operations engineers. But how do you separate and manage these workloads so that users get the best experience while minimizing costs?

In this post, I show you how to use workgroups to do the following:

• Separate workloads.
• Control user access.
• Manage query usage and costs.

Separate workloads

By default, all Athena queries execute in the primary workgroup.  As an administrator, you can create new workgroups to separate different types of workloads.  Administrators commonly turn to workgroups to separate analysts running ad hoc queries from automated reports.  Here’s how to build out that separation.

First create two workgroups, one for ad hoc users (ad-hoc-users) and another for automated reports (reporting).

Next, select a specific output location. All queries executed inside this workgroup save their results to this output location. Routing results to a single secure location helps make sure users only access data they are permitted to see. You can also enforce encryption of query results in S3 by selecting the appropriate encryption configuration.

Workgroups also help you simplify the onboarding of new users to Athena. By selecting override client-side settings, you enforce a predefined configuration on all queries within a workgroup. Users no longer have to configure a query results output location or S3 encryption keys. These settings default to the parameters defined for the workgroup where those queries execute. Additionally, each workgroup maintains a unique query history and saved query inventory, making queries easier for you to track down.

Finally, when creating a workgroup, you can add up to 50 key-value pair tags to help identify your workgroup resources. Tags are also useful when attempting to allocate Athena costs between groups of users. Create Name and Dept tags for the ad-hoc-users and reporting workgroups with their name and department association.

Control user access to workgroups

Now that you have two workgroups defined, ad-hoc-users and reporting, you must control who can use and update them.  Remember that workgroups are IAM resources and therefore have an ARN. You can use this ARN in the IAM policy that you associate with your users.  In this example, create a single IAM user representing the team of ad hoc users and add the individual to an IAM group. The group contains a policy that enforces what actions these users can perform.

Start by reviewing IAM Policies for Accessing Workgroups and Workgroup Example Policies to familiarize yourself with policy options. Use the following IAM policy to set up permissions for your analyst user. Grant this user only the permissions required for working in the ad-hoc-users workgroup. Make sure that you tweak this policy to match your exact needs:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "athena:ListWorkGroups"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "athena:StartQueryExecution",
                "athena:GetQueryResults",
                "athena:DeleteNamedQuery",
                "athena:GetNamedQuery",
                "athena:ListQueryExecutions",
                "athena:StopQueryExecution",
                "athena:GetQueryResultsStream",
                "athena:GetQueryExecutions",
                "athena:ListNamedQueries",
                "athena:CreateNamedQuery",
                "athena:GetQueryExecution",
                "athena:BatchGetNamedQuery",
                "athena:BatchGetQueryExecution",
                "athena:GetWorkGroup",
                "athena:ListTagsForResource"
            ],
            "Resource": "arn:aws:athena:us-east-1:112233445566:workgroup/ad-hoc-users"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObjectAcl",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": "arn:aws:s3:::demo/workgroups/adhocusers/*"
        },
{
            "Effect": "Allow",
            "Action": [
                "glue:Get*"
            ],
            "Resource": [
                "arn:aws:glue:us-east-1:112233445566:catalog",
                "arn:aws:glue:us-east-1:112233445566:database/amazon",
                "arn:aws:glue:us-east-1:112233445566:table/amazon/*"
            ]
        }
    ]
}

Now your analyst user can execute queries only in the ad-hoc-users workgroup. The analyst user can switch to other workgroups, but they lose access when they try to perform any action. They are further restricted to list and query only those tables that belong to the Amazon database. For more information about controlling access to AWS Glue resources such as databases and tables, see AWS Glue Resource Policies for Access Control.

The following screenshot shows what the analyst user sees in the Athena console:

I’ve created a simple Node.js tool that executes SQL queries stored as files in a given directory. You can find my Athena test runner code in the athena_test_runner GitHub repo. You can use this code to simulate a reporting tool, after configuring it to use a workgroup. To do that, create an IAM role with permissions like those previously defined for the analyst user. This time, restrict access to the reporting workgroup.

The following JavaScript code example shows how to select a workgroup programmatically when executing queries:

function executeQueries(files) {
    params = 
    {
      "QueryString": "", 
      "ResultConfiguration": { 
        "OutputLocation": ""
      },
      "QueryExecutionContext": {
        "Database": "default"
      },
      "WorkGroup":"reporting"
    }
 
    params.QueryString = "SELECT * FROM amazon.final_parquet LIMIT 10"
    return new Promise((resolve, reject) => {
        athena.startQueryExecution(params, (err, results) => {
            if (err) {
                reject(err.message)
            } else {
                resolve(results)
            }
        })
    })
}

Run sample automated reports under the reporting workgroup, with the following command:

node index.js testsuite/

Query histories remain isolated between workgroups. A user logging into the Athena console as an analyst using the ad-hoc-users workgroup doesn’t see any automated reports that you ran under the reporting workgroup.

Managing query usage and cost

You have two workgroups configured: one for ad hoc users and another for automated reports. Now, you must safeguard against bad queries. In this use case, two potential situations for query usage should be monitored and controlled:

• Make sure that users don’t run queries that scan more data than allowed by their budget.
• Safeguard against automated script bugs that could cause indefinite query retirement.

First, configure data usage controls for your ad-hoc-users workgroup. There are two types of data usage controls: per-query and per-workgroup.

Set the per-query control for analysts to be 1 GB. This control cancels any query run in the ad-hoc-users workgroup that tries to scan more than 1 GB.

To observe this limit in action, choose Update, return to the query editor, and run a query that would scan more than 1 GB. This query triggers the error message, “Query cancelled! : Bytes scanned limit was exceeded”. Remember that you incur charges for data the query scanned up to the point of cancellation. In this case, you incur charges for 1 GB of data.

Now, switch to your reporting workgroup. For this workload, you’re not worried about individual queries scanning too much data. However, you want to control the aggregate amount of data scanned of all queries in this workgroup.

Create a per-workload data usage control for the reporting workgroup. You can configure the maximum amount of data scanned by all queries in the workgroup during a specific period.

For the automated reporting workload, you probably have a good idea of how long the process should take and the total amount of data that queries scan during this time. You only have a few reports to run, so you can expect them to run in a few minutes, only scanning a few megabytes of data. Begin by setting up a low watermark alarm to notify you when your queries have scanned more data than you would expect in five minutes. The following example is for demo purposes only. In most cases, this period would be longer. I configured the alarm to send a notification to an Amazon SNS topic that I created.

To validate the alarm, I made a minor change to my test queries, causing them to scan more data. This change triggered the SNS alarm, shown in the following Amazon CloudWatch dashboard:

Next, create a high watermark alarm that is triggered when the queries in your reporting workgroup exceed 1 GB of data over 15 minutes. In this case, the alarm triggers an AWS Lambda function that disables the workgroup, making sure that no additional queries execute in it. This alarm protects you from incurring faulty automation code or runaway query costs.

Before creating the data usage control, create a Node.js Lambda function to disable the workgroup. Paste in the following code:

exports.handler = async (event) => {
    const AWS = require('aws-sdk')
    let athena = new AWS.Athena({region: 'us-east-1'})
 
    let msg = JSON.parse(event.Records[0].Sns.Message)
    let wgname = msg.Trigger.Dimensions.filter((i)=>i.name=='WorkGroup')[0].value
    
    athena.updateWorkGroup({WorkGroup: wgname, State: 'DISABLED'})
 
    const response = {
        statusCode: 200,
        body: JSON.stringify(`Workgroup ${wgname} has been disabled`),
    };
    return response;
}

This code grabs the workgroup name from the SNS message body and calls the UpdateWorkGroup API action with the name and the state of DISABLED. The Athena API requires the most recent version of the AWS SDK. When you create the Lambda bundle, include the latest AWS SDK version in that bundle.

Next, create a new SNS topic and a subscription. For Protocol, select AWS Lambda. Then, select the Lambda function that you created in the previous step.

In the Athena console, create the second alarm, 1 GB for 15 min., and point it to the SNS topic that you created earlier. When triggered, this SNS topic calls the Lambda function that disables the reporting workgroup. No more queries can execute in this workgroup. You see this error message in the console when a workgroup is disabled:

Athena exposes other aggregated metrics per workgroup under the AWS/Athena namespace in CloudWatch, such as the query status and the query type (DDL or DML) per workgroup. To learn more, see Monitoring Athena Queries with CloudWatch Metrics.

Cost allocation tags

When you created your ad-hoc-users and reporting workgroups, you added Name and Dept tags. These tags can be used in your Billing and Cost Management console to determine the usage per workgroup.

Summary

In this post, you learned how to use workgroups in Athena to isolate different query workloads, manage access, and define data usage controls to protect yourself from runaway queries. Metrics exposed to CloudWatch help you monitor query performance and make sure that your users are getting the best experience possible. For more details, see Using Workgroups to Control Query Access.

About the Author

Roy Hasson is a Global Business Development Manager for AWS Analytics. He works with customers around the globe to design solutions to meet their data processing, analytics and business intelligence needs. Roy is big Manchester United fan, cheering his team on and hanging out with his family.

Feed: AWS Big Data Blog.

Amazon always works backwards from the customer’s needs. Customers have made strong requests that they want stored procedures in Amazon Redshift, to make it easier to migrate their existing workloads from legacy, on-premises data warehouses.

With that primary goal in mind, AWS chose to implement PL/pqSQL stored procedure to maximize compatibility with existing procedures and simplify migrations. In this post, we discuss how and where to use stored procedures to improve operational efficiency and security. We also explain how to use stored procedures with AWS Schema Conversion Tool.

What is a stored procedure?

A stored procedure is a user-created object to perform a set of SQL queries and logical operations. The procedure is stored in the database and is available to users who have sufficient privileges to run it.

Unlike a user-defined function (UDF), a stored procedure can incorporate data definition language (DDL) and data manipulation language (DML) in addition to SELECT queries. A stored procedure doesn’t have to return a value. You can use the PL/pgSQL procedural language, including looping and conditional expressions, to control logical flow.

Stored procedures are commonly used to encapsulate logic for data transformation, data validation, and business-specific operations. By combining multiple SQL steps into a stored procedure, you can reduce round trips between your applications and the database.

You can also use stored procedures for delegated access control. For example, you can create stored procedures to perform functions without giving a user access to the underlying tables.

Why would you use stored procedures?

Many customers migrating to Amazon Redshift have complex data warehouse processing pipelines built with stored procedures on their legacy data warehouse platform. Complex transformations and important aggregations are defined with stored procedures and reused in many parts of their processing. Re-creating the logic of these processes using an external programming language or a new ETL platform could be a large project. Using Amazon Redshift stored procedures allows you to migrate to Amazon Redshift more quickly.

Other customers would like to tighten security and limit the permissions of their database users. Stored procedures offer new options to allow DBAs to perform necessary actions without having to give permissions too widely. With the security definer concepts in stored procedures, it is now possible to allow users to perform actions they otherwise would not have permissions to run.

Additionally, using stored procedures in this way helps reduce the operations burden. An experienced DBA is able to define a well-tested process for some administrative or maintenance action. They can then allow other, less experienced operators to execute the process without entrusting them with full superuser permissions on the cluster.

Finally, some customers prefer using stored procedures to manage their ETL/ELT operations as an alternative to shell scripting or complex orchestration tools. It can be difficult to ensure that shell scripts correctly retrieve and interpret the state of each operation in an ETL/ELT process. It can also be challenging to take on the operation and maintenance of an orchestration tool with a small data warehouse team.

Stored procedures allow the ETL/ELT logical steps to be fully enclosed in a master procedure that is written so that it either succeeds completely or fails cleanly with no side effects. The stored procedure can be called with confidence from a simple scheduler like cron.

Create a stored procedure

To create a stored procedure in Amazon Redshift, use the following the syntax:

CREATE [ OR REPLACE ] PROCEDURE sp_procedure_name 
  ( [ [ argname ] [ argmode ] argtype [, ...] ] )
AS $$
  procedure_body
$$ LANGUAGE plpgsql 
[ { SECURITY INVOKER | SECURITY DEFINER } ]
[ SET configuration_parameter { TO value | = value } ]

When you design stored procedures, think about the encapsulated functionality, input and output parameters, and security level. As an example, here’s how you can write a stored procedure to check primary key violations, given names of the schema, table, and primary key column, using dynamic SQL:

CREATE OR REPLACE PROCEDURE check_primary_key(schema_name varchar(128), 
table_name varchar(128), col_name varchar(128)) LANGUAGE plpgsql
AS $$
DECLARE
  cnt_var integer := 0;
BEGIN
  SELECT INTO cnt_var count(*) from pg_table_def where schemaname = schema_name and
  tablename = table_name and "column" = col_name;
  IF cnt_var = 0 THEN
    RAISE EXCEPTION 'Input table or column does not exist.';
  END IF;

  DROP TABLE IF EXISTS duplicates;
  EXECUTE
    $_$ CREATE TEMP TABLE duplicates as
    SELECT $_$|| col_name ||$_$, count(*) as counter
    FROM $_$|| table_name ||$_$
    GROUP BY 1
    HAVING count(*) > 1
    ORDER BY counter desc $_$;
  SELECT INTO cnt_var COUNT(*) FROM duplicates;
  IF cnt_var = 0
    THEN RAISE INFO 'No duplicates found';
    DROP TABLE IF EXISTS duplicates;
  ELSE
    RAISE INFO 'Duplicates exist for % value(s) in column %', cnt, col_name;
    RAISE INFO 'Check tmp table "duplicates" for duplicated values';
  END IF;
END;
$$;

For details about the kinds of SQL queries and control flow logic that can be used inside a stored procedure, see Creating Stored Procedures in Amazon Redshift.

Invoke a stored procedure

Stored procedures must be invoked by the CALL command, which takes the procedure name and the input argument values. CALL can’t be part of any regular queries. As an example, here’s how to invoke the stored procedure created earlier:

db=# call check_primary_key('public', 'bar', 'b');
INFO:  Duplicates exist for 1 value(s) in column b
INFO:  Check tmp table "duplicates" for duplicated values

Amazon Redshift stored procedure calls can return results through output parameters or a result set. Nested and recursive calls are also supported. For details, see CALL command.

How to use security definer procedures

Now that you know how to create and invoke a stored procedure, here’s more about the security aspects. When you create a procedure, only you as the owner (creator) have the privilege to call or execute it. You can grant EXECUTE privilege to other users or groups, which enables them to execute the procedure. EXECUTE privileges do not automatically imply that the caller can access all database objects (tables, views, and so on) that are referenced in the stored procedure.

Take the example of a procedure, sp_insert_customers, created by user Mary. It has an INSERT statement that writes to table customers that is owned by Mary. If Mary grants EXECUTE privileges to user John, John still can’t INSERT into the table customers unless he has explicitly been granted INSERT privileges on customers.

However, it might make sense to allow John to call the stored procedure without giving him INSERT privileges on customers. To do this, Mary has to set the SECURITY attribute of the procedure to DEFINER when creating the procedure and then grant EXECUTE privileges to John. With this set, when John calls sp_insert_customers, it executes with the privileges of Mary and can insert into customers without him having been granted INSERT privileges on that table.

When the security attribute is not specified during procedure creation, its value is set to INVOKER by default. This means that the procedure executes with the privileges of the user that calls it. When the security attribute is explicitly set to DEFINER, the procedure executes with the privileges of the procedure owner.

Best practices with stored procedures in Amazon Redshift

Here are some best practices for using stored procedures.

Ensure that stored procedures are captured in your source control tool.

If you plan to use stored procedures as a key element of your data processing, you should also establish a practice of committing all stored procedure changes to a source control system.

You could also consider defining a specific user who is the owner of important stored procedures and automating the process of creating and modifying procedures.

You can retrieve the source for existing stored procedures using the following command:

SHOW procedure_name;

Consider the security scope of each procedure and who calls it

By default, stored procedures run with the permission of the user that calls them. Use the SECURITY DEFINER attribute to enable stored procedures to run with different permissions. For instance, explicitly revoke access to DELETE from an important table and define a stored procedure that executes the delete after checking a safe list.

When using SECURITY DEFINER, take care to:

• Grant EXECUTE on the procedure to specific users, not to PUBLIC. This ensures that the procedure can’t be misused by general users.
• Qualify all database objects that the procedure accesses with the schema names if possible. For example, use myschema.mytable instead of just mytable.
• Set the search_path when creating the procedure by using the SET option. This prevents objects in other schemas with the same name from being affected by an important stored procedure.

Use set-based logic and avoid manually looping over large datasets

When manipulating data within your stored procedures, continue to use normal, set-based SQL as much as possible, for example, INSERT, UPDATE, DELETE.

Stored procedures provide new control structures such as FOR and WHILE loops. These are useful for iterating over a small number of items, such as a list of tables. However, you should avoid using the loop structures to replace a set-based SQL operation. For example, iterating over millions of values to update them one-by-one is inefficient and slow.

Be aware of REFCURSOR limits and use temp tables for larger result sets

Result sets may be returned from a stored procedure either as a REFCURSOR or using temp tables.  REFCURSOR is an in-memory data structure and is the simplest option in many cases.

However, there is a limit of one REFCURSOR per stored procedure. You may want to return multiple result sets, interact with results from multiple sub-procedures, or return millions of result rows (or more). In those cases, we recommend directing results to a temp table and returning a reference to the temp table as the output of your stored procedure.

Keep procedures simple and nest procedures for complex processes

Try to keep the logic of each stored procedure as simple possible. You maximize your flexibility and make your stored procedures more understandable by keeping them simple.

The code of your stored procedures can become complex as you refine and enhance them. When you encounter a long and complex stored procedure, you can often simplify by moving sub-elements into a separate procedure that is called from the original procedure.

Migrating a stored procedure with the AWS Schema Conversion Tool

With Amazon Redshift announcing the support for stored procedures, AWS also enhanced AWS Schema Conversion Tool to convert stored procedures from legacy data warehouses to Amazon Redshift.

AWS SCT already supports the conversion of Microsoft SQL Server data warehouse stored procedures to Amazon Redshift.

With build 627, AWS SCT can now convert Microsoft SQL Server data warehouse stored procedures to Amazon Redshift. Here are the steps in AWS SCT:

1. Create a new OLAP project for a SQL Server data warehouse (DW) to Amazon Redshift conversion.
2. Connect to the SQL Server DW and Amazon Redshift endpoints.
3. Uncheck all nodes in the source tree.
4. Open the context (right-click) menu for Schemas.
5. Open the context (right-click) menu for the Stored Procedures node and choose Convert Script (just like when you convert database objects).
6. (Optional) You can also choose to review the assessment report and apply the conversion.

Here is an example of a SQL Server DW stored procedure conversion:

Conclusion

Stored procedure support in Amazon Redshift is now generally available in every AWS Region. We hope you are as excited about running stored procedures in Amazon Redshift as we are.

With stored procedure support in Amazon Redshift and AWS Schema Conversion Tool, you can now migrate your stored procedures to Amazon Redshift without having to encode them in another language or framework. This feature reduces migration efforts. We hope more on-premises customers can take advantage of Amazon Redshift and migrate to the cloud for database freedom.

About the Authors

Joe Harris is a senior Redshift database engineer at AWS, focusing on Redshift performance. He has been analyzing data and building data warehouses on a wide variety of platforms for two decades. Before joining AWS he was a Redshift customer from launch day in 2013 and was the top contributor to the Redshift forum.

Abhinav Singh is a database engineer at AWS. He works on design and development of database migration projects as well as customers to provide guidance and technical assistance on database migration projects, helping them improve the value of their solutions when using AWS.

Entong Shen is a senior software engineer on the Amazon Redshift query processing team. He has been working on MPP databases for over 7 years and has focused on query optimization, statistics and SQL language features. In his spare time, he enjoys listening to music of all genres and working in his succulent garden.

Vinay is a principal product manager at Amazon Web Services for Amazon Redshift. Previously, he was a senior director of product at Teradata and a director of product at Hortonworks. At Hortonworks, he launched products in Data Science, Spark, Zeppelin, and Security. Outside of work, Vinay loves to be on a yoga mat or on a hiking trail.
 

Sushim Mitra is a software development engineer on the Amazon Redshift query processing team. He focuses on query optimization problems, SQL Language features and Database security. When not at work, he enjoys reading fiction from all over the world.

Feed: Planet MySQL
;
Author: MySQL Performance Blog
;

In this blog post, we will discuss a new feature – the MySQL 8.0.17 clone plugin. Here I will demonstrate how easy it is to use to create the “classic” replication, building the standby replica from scratch.

The clone plugin permits cloning data locally or from a remote MySQL server instance. The cloned data is a physical snapshot of data stored in InnoDB, and this means, for example, that the data can be used to create a standby replica.

Let’s go to the hands-on and see how it works.

Installation & validation process of the MySQL 8.0.17 clone plugin

Installation is very easy and it works in the same as installing other plugins. Below is the command line to install the clone plugin:

And how to check if the clone plugin is active:

Note that these steps need to be executed on the Donor (aka master) and on the Recipient (aka slave if the clone is being used to create a replica).

After executing the installation, the plugin will be loaded automatically across restarts, so you don’t need to worry about this anymore.

Next, we will create the user with the necessary privilege on the Donor, so we can connect to the instance remotely to clone it.

As a security measure, I recommend replacing the % for the IP/hostname or network mask of the Recipient so the connections will be accepted only by the future replica server.  Now, on the Recipient server, the clone user requires the CLONE_ADMIN privilege for replacing recipient data, blocking DDL during the cloning operation and automatically restarting the server.

Next, with the plugin installed and validated, and users created on both Donor and Recipient servers, let’s proceed to the cloning process.

Cloning process

As mentioned, the cloning process can be executed locally or remotely.  Also, it supports replication, which means that the cloning operation extracts and transfers replication coordinates from the donor and applies them on the recipient. It can be used for GTID or non-GTID replication.

So, to begin the cloning process, first, let’s make sure that there’s a valid donor. This is controlled by clone_valid_donor_list parameter. As it is a dynamic parameter, you can change it while the server is running. Using the show variables command will show if the parameter has a valid donor:

In our case, we need to set it. So let’s change it:

The next step is not mandatory, but using the default log_error_verbosity the error log does not display much information about the cloning progress. So, for this example, I will adjust the verbosity to a higher level (on the Donor and the Recipient):

Now, let’s start the cloning process on the Recipient:

It is possible to observe the cloning progress in the error log of both servers. Below is the output of the Donor:

And the Recipient:

Note that the MySQL server on the Recipient will be restarted after the cloning process finishes. After this, the database is ready to be accessed and the final step is setting up the replica.

The replica process

Both binary log position (filename, offset) and GTID coordinates are extracted and transferred from the donor MySQL server instance.

The queries below can be executed on the cloned MySQL server instance to view the binary log position or the GTID of the last transaction that was applied:

With the information in hand, we need to execute the CHANGE MASTER command accordingly:

Limitations

The clone plugin has some limitations and they are described here. In my opinion, two major limitations will be faced by the community. First, it is the ability to clone only InnoDB tables. This means that  MyISAM and CSV tables stored in any schema including the sys schema will be cloned as empty tables.

The other limitation is regarding DDL, including TRUNCATE TABLE, which is not permitted during a cloning operation. Concurrent DML, however, is permitted. If a DDL is running, then the clone operation will wait for the lock:

Otherwise, if the clone operation is running, the DDL will wait for the lock:

Conclusion

Creating replicas has become much easier with the help of the MySQL 8.0.17 clone plugin. This feature can be used using SSL connections and with encrypted data as well.  At the moment of publication of this blog post, the clone plugin can be used to set up not only asynchronous replicas but provisioning Group Replication members too. Personally, I believe that in the near future this feature will also be used for Galera clusters. This is my two cents for what the future holds.

Useful Resources

Finally, you can reach us through the social networks, our forum or access our material using the links presented below:

Feed: Planet MySQL
;
Author: Valeriy Kravchuk
;

After a 3 months long break I’d like to continue reviewing MySQL bug reports that I am subscribed to. This issue is devoted to bug reports I’ve considered interesting to follow in May, 2019:

• Bug #95215 – “Memory lifetime of variables between check and update incorrectly managed“. As demonstrated by Manuel Ung, there is a problem with all InnoDB MYSQL_SYSVAR_STR variables that can be dynamically updated. Valgrind allows to highlight it.
• Bug #95218 – “Virtual generated column altered unexpectedly when table definition changed“. This weird bug (that does not seem to be repeatable on MariaDB 10.3.7 with proper test case modifications like removing NOT NULL and collation settings from virtual column) was reported by Joseph Choi. Unfortunately we do not see any documented attempt to check if MySQL 8.0.x is also affected. My quick test shows MySQL 8.0.17 is NOT affected, but I’d prefer to see check copy/pasted as a public comment to the bug.
• Bug #95230 – “SELECT … FOR UPDATE on a gap in repeatable read should be exclusive lock“. There are more chances to get a deadlock with InnoDB than one might expect… I doubt this report from Domas Mituzas is a feature request. It took him some extra efforts to insist on the point and get it verified even as S4.
• Bug #95231 – “LOCK=SHARED rejected contrary to specification“. This bug report from Monty Solomon ended up as a documentation request. The documentation and the implementation are not aligned, and it was decided NOT to change the parser to match documented syntax. But why it is still “Verified” then? Should it take months to correct the fine manual?
• Bug #95232 – “The text of error message 1846 and the online DDL doc table should be updated”. Yet another bug report from Monty Solomon. Some (but not ALL) partition specific ALTER TABLE operations do not yet support LOCK clause.
• Bug #95233 – “check constraint doesn’t consider IF function that returns boolean a boolean fun“. As pointed out by Daniel Black, IF() function in a check constraint isn’t considered a boolean type. He had contributed a patch to fix this, but based on comments it’s not clear if it’s going to be accepted and used “as is”. The following test shows that MariaDB 10.3 is not affected:
  

  C:Program FilesMariaDB 10.3bin>mysql -uroot -proot -P3316 test
  Welcome to the MariaDB monitor.  Commands end with ; or g.
  Your MariaDB connection id is 9
  Server version: 10.3.7-MariaDB-log mariadb.org binary distribution

  Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

  Type ‘help;’ or ‘h’ for help. Type ‘c’ to clear the current input statement.
  MariaDB [test]> create table t1 (source enum(‘comment’,’post’) NOT NULL, comment_id int unsigned, post_id int unsigned);
  Query OK, 0 rows affected (0.751 sec)

  MariaDB [test]> alter table t1 add check(IF(source = ‘comment’, comment_id IS NOT NULL AND post_id IS NULL, post_id IS NOT NULL AND comment_id IS NULL));
  Query OK, 0 rows affected (1.239 sec)
  Records: 0  Duplicates: 0  Warnings: 0

• Bug #95235 – “ABRT:Can’t generate a unique log-filename binlog.(1-999), while rotating the bin“. Yet another bug report from Daniel Black. When MySQL 8.0.16 is built with gcc 9.0.x abort is triggered in the MTR suite on the binlog.binlog_restart_server_with_exhausted_index_value test.
• Bug #95249 – “stop slave permanently blocked“. This bug was reported by Wei Zhao, who had contributed a patch.
• Bug #95256 – “MySQL 8.0.16 SYSTEM USER can be changed by DML“. MySQL 8.0.16 had introduced an new privilege, SYSTEM_USER. MySQL manual actually says:
  

  “The protection against modification by regular accounts that
  is afforded to system accounts by the
  SYSTEM_USER privilege does
  not apply to regular accounts that have privileges on the
  mysql system schema and thus can directly
  modify the grant tables in that schema. For full protection,
  do not grant mysql schema privileges to
  regular accounts.“

  But the report that a user with a privilege to execute DML on the mysql.GLOBAL_GRANTS table from Zhao Jianwei was accepted and verified. I hope Oracle engineers will finally make up their mind and decide either to fix this or to close this report as “Not a bug”. I’ve subscribed in a hope for some fun around this decision making.

• Bug #95269 – “binlog_row_image=minimal causes assertion failure“. This assertion failure happens in debug build when one of standard MTR test cases, rpl.rpl_gis_ddl or rpl.rpl_window_functions is executed with –binlog-row-image=minimal option. In such cases I always wonder what is the reason for a failure NOT to be noted by Oracle MySQL QA and somehow fixed before Community users notice it? Either they don’t run tests on debug builds with all possible combinations, or do not care to fix such failures (and thus should suffer from known failures in other test runs). I do not like any of these options, honestly. The bug was reported by Song Libing.
• Bug #95272 – “Potential InnoDB SPATIAL INDEX corruption during root page split“. This bug was reported by Albert Hu based on Valgrind report when running the test innodb.instant_alter. Do they run MTR tests under on Valgrind or ASan builds in Oracle? I assume they do, but then why Community users are reporting such cases first? Note that related MariaDB’s bug, MDEV-13942, is fixed in 10.2.24+ and 10.3.15+.
• Bug #95285 – “InnoDB: Page [page id: space=1337, page number=39] still fixed or dirty“. This assertion failure that happens during normal shutdown was reported by LUCA TRUFFARELLI. There are chances that this is a regression bug (without a regression tag), as it does not happen for reporter on MySQL 5.7.21.
• Bug #95319 – “SHOW SLAVE HOST coalesces certain server_id’s into one“. This bug was reported by Lalit Choudhary from Percona based on original findings by Glyn Astill.
• Bug #95416 – “ZERO Date is both NULL and NOT NULL“. This funny bug report was submitted Morgan Tocker. Manual actually explains that it’s intended behavior (MariaDB 10.3.7 works the same way as MySQL), but it’s still funny and unexpected, and the bug report remains “Verified”.
• Bug #95478 – “CREATE TABLE LIKE does not honour ROW_FORMAT.” I’d like to add “…when it was not defined explicitly for the original table”. The problem was reported by Jean-François Gagné and ended up as a verified feature request. See also this my post on the details of where row_format is stored and is not stored for InnoDB tables…
• Bug #95484 – “EXCHANGE PARTITION works wrong/werid with different ROW_FORMAT“. Another bug report by Jean-François Gagné related to the previous one. He had shown that it’s actually possible to get partitions with different row formats in the same InnoDB table in MySQL 5.7.26, but not in the most natural way. It seems the problem may be fixed in 5.7.27 (by the fix for another, internally reported bug), but the bug remains “Verified”.

There are some more bugs reported in May 2019 that I was interested in, but let me stop for now. Later in May I’ve got a chance to spend some days off in Barcelona, without any single MySQL bug report opened for day.

I like this view of Barcelona way more than any MySQL bugs review, including this one.

To summarize:

1. Oracle engineers who process bugs still sometimes do not care to check if all
   supported major versions are affected and/or share the results of such
   checks in public. Instead, some of them care to argue about severity of the bug report, test case details etc.
2. We still see bug reports that originates from existing, well known MTR test cases runs under Valgrind or in debug builds with some non-default options set. I do not have any good reason in mind to explain why these are NOT reported by Oracle’s internal QA first.
3. Surely some regression bugs still get verified without the regression tag added.

I truly hope my talk “Problems with Oracle’s Way of MySQL Bugs Database Maintenance” will be accepted for Percona Live Europe 2019 conference (at least as a lightning talk) and I’ll get another chance to speak about the problems highlighted above, and more. There are some “metabugs” in the way Oracle handles MySQL bug report, and these should be discussed and fixed, for the benefits of MySQL quality and all MySQL users and customers.

Feed: Teradata Blog.

What is Self-Service Analytics?

Self-service analytics is a common buzzword for many organizations which desire to be more data driven and less dependent on IT for their data needs. Organizations are increasingly implementing self-service capabilities to enable and promote a data-driven culture within their organizations. Gartner, Inc. predicts that by 2019, the analytics output of business users with self-service capabilities will surpass that of professional data scientists. Self-service analytics sound great, but how sustainable is it? In a series of articles, we will dive into self-service, starting first with “What is self-service analytics?”.  

Every user wants to be self-service enabled, but not all users have the same skillset. Therefore, enabling self-service means different things to different users. Let’s first start by defining some basic user personas, their skillsets and what self-service means to each of them.
 

• General Consumer – Analytic users who leverage reports and dashboards from trusted, governed data sources to run the day-to-day business. They work exclusively from an operational perspective. They leverage BI tools and dashboards to monitor and analyze the data.
• Data Analyst – Analytic users who run reports, manipulate or pivot on prepared data products. Access rules are usually restricted to prepared data products from trusted, governed data sources, which may be optimized structures, sometimes referencing common summaries. They primarily work from an operational perspective but do some data discovery as well, when needed. They may leverage the same BI tools as the general consumer but are also savvy at writing SQL code. They can directly query data which has been curated to a tabular structure, e.g. a relational database schema, or a schema defined by Hive or some other interface that enables data queries. They are skilled spreadsheet users.
• Citizen Data Scientist – Data analyst which can query a broad range of data products looking for answers to new business problems or discover new insights from existing data which is trusted and governed. They may also leverage new experimental data. They work primarily from an R&D perspective. These individuals typically have a strong, business subject area knowledge, SQL coding skills and an understanding of how to apply statistical and machine learning algorithms as a black box in solving business problems.
• Data Scientist – Work with any data source (raw or curated) to look for answers to new business problems or discover new insights. They will bring their own data and augment it with existing data as needed. They most often work from a R&D perspective. They have the most diverse skill set, e.g. strong interpretive language skills (e.g. Python, R) and SQL skills, and solid mathematic skills for leveraging machine and deep learning algorithms (often a Ph.D. in science or mathematics). They might be a business subject matter expert or work with a business subject matter expert to better understand the business problem they are trying to solve.

These personas are not absolute, but the generalized categorizations enable us to group like users by their skillsets, needs and data reach within the analytic ecosystem. 

General consumers reach into the Access Layer (typically star schemas) via OLAP tools and dashboards. Data analysts leverage the Access Layer and reach further into the Integration Layer (typically a near 3^rd normal form structure) to do ad-hoc analytics. Citizen data scientists leverage both the Access and Integration Layers, but also leverage standardized data (typically flattened structures, but not necessarily) in the Acquisition Layer. The data scientists, with their unique skillset, have full access to all forms of data within the entire analytics ecosystem. All users will leverage trusted and governed data, where possible, as it possesses the highest data quality and requires the least effort to utilize.

All personas, except the general consumers, need to provision resources for their self-service analytics. These terms can mean something different to each persona and are described below to provide a more consistent definition in support of managing self-service analytics.

Automated-Provisioning: The goal should be to enable users to request resources which can be systematically enabled within sanctioned areas, if they fall within standard defaults, e.g. storage, compute, memory. An exploration zone, such as Teradata Data Labs, is a good example of this capability in action.  

Self-Service Analytics: The ability for the user to access and utilize available resources (e.g. storage, compute, memory) so that they can acquire, profile, wrangle and analyze data (structured or unstructured) for some analytical purpose on their own. This could be a data scientist doing advanced analytics or a data analyst running SQL and NoSQL queries or connecting an OLAP tool for investigating new reports and dashboards. They may or may not require new data. By far the largest segment of self-service analytics are the general consumers, leveraging BI tools to slice and dice within the tool’s defined data domain.    

Individuals doing self-service analytics should be free to do what they need by themselves. They read, seek out online training, attend instructor led training classes, etc. They learn as they go, most often when they need to remove a roadblock. Publishing and maintaining standards and best practices further enhances self-paced learning.

Some users will need to load data into a relational database, which means writing DDL code, either by using command line or GUI tools. A data analyst is very familiar with SQL, so creating a table and loading it should not be a stretch. A citizen data scientist may also be able to load data into NoSQL structures and perhaps wrangle and conform some data into a usable format. The data scientist can do pretty much everything needed to load and prepare any type of data.  

At some point, exploration results and model definition techniques that both lead to reuse and faster time-to-value are important enough to be recalculated and reanalyzed on a regular basis and should become part of the organization’s critical production processing. Operationalization is the process by which data and/or analytic modules are moved from an exploration environment into a production environment. It entails adhering to data naming conventions, optimizing processes and utilizing standard services, thereby ensuring the data and analytics will be consistently available to the entire user community. Perhaps even more important, it frees up users to focus on other research needs.

Everyone wants to speed up self-service analytics. Depending on the choices made during exploration, the operationalization effort may take longer. A common example would be using tools for exploration, which are not authorized in production. The more aware the users are, the better they understand the trade-offs.  

There is a lot of hype around self-service analytics. The complexity goes up when a user decides to bring their own data; therefore, making data easier to find and access will greatly accelerate self-service analytics initiatives and minimize data sprawl.

There is a lot of hype around self-service analytics. The complexity goes up when a user decides to bring their own data; therefore, making data easier to find and access will greatly accelerate self-service analytics initiatives and minimize data sprawl.

In part 2 of this series we will discuss the value and challenges of self-service analytics within organizations today.

Feed: Planet MySQL
;
Author: Valeriy Kravchuk
;

It’s Saturday night. I have a week of vacation ahead that I am going to spend at home, working on things I usually do not have time for. I already did something useful (created a couple of test cases for MariaDB bugs here and there, among other things), so why not to get some fun and continue my previous review of recent interesting MySQL bug report…

So, here is the list of 15 more MySQL community bug reports that I’ve subscribed to back in May and June 2019:

• Bug #95491 – “The unused wake up in simulator AIO of function reserve_slot“. I am impressed by the fact that these days many people just study InnoDB code, find problematic parts, improve them, test and prove the improvement, and submit patches to Oracle. Chen Zongzhi, who reported this bug, is one of such great people.
• Bug #95496 – “Replication of memory tables“. As pointed out by Iwo P, MySQL manual describes what happens to MEMORY tables replicated from master to slave in details, including the case of master restart. But one detail is missing – when slave with MEMORY tables restarts, even with super_read_only set, it also generates even to delete data from the table into its own binary log, with GTID, and this leads to problems in case of failover.
• Bug #95507 – “innodb_stats_method is not honored when innodb_stats_persistent=ON“. This bug was reported by my colleague Sergei Petrunia. See also related MDEV-19574  (still “Open”).
• Bug #95547 – “MySQL restarts automatically after ABORT_SERVER due to disk full“. I am not sure this is a replication bug, more a problem of too primitive mysqld_safe or systemd startup scripts that do not check free space on disk or reason why mysqld process committed suicide. Anyway, this report by Przemyslaw Malkowski from Percona was accepted as a replication bug.
• Bug #95582 – “DDL using bulk load is very slow under long flush_list“. As found and proved by Fungo Wang, due to optimized away redo log writing for bulk load, to make sure such DDL is crash safe, dirtied pages must be flushed to disk synchronously. Currently this is done inefficiently, as shown in this bug report. See also a year old (and still “Open”) Bug #92099 – “provide an option to record redo log for bulk loading” from the same reporter, where he asks for a way to get redo logging back (but still benefit from other “online” ALTER features). MariaDB has that option after the fix of MDEV-16809.
• Bug #95612 – “replication breakage due to Can not lock user management caches for processing“. Simon Mudd noted that in some cases simple DROP USER IF EXISTS … breaks parallel replication in MySQL 8.0.16. Good to know that START SLAVE allows to continue
• Bug #95698 – “JSON columns are returned with binary charset“. This bug was reported by Daniel Book. Note that according to the original issue, DBD::MariaDB (I had no idea this exists!) works properly with MariaDB 10.3 or 10.4. I had not checked.
• Bug #95734 – “Failing SET statement in a stored procedure changes variables“. This really weird and unexpected bug was reported by Przemysław Skibiński.
• Bug #95863 – “MTS ordered commit may cause deadlock “. Great finding by Lalit Choudhary from Percona!
• Bug #95895 – “Shutdown takes long time when table uses AUTOINC and FOREIGN KEY constraints“. It was reported by Satya Bodapati from Percona, who later contributed as patch. See also his related Bug #95898 – “InnoDB releases memory of table only on shutdown if table has FK constraint”. I truly hope to see these fixed in next minor releases of MySQL. Note that originally bug (PS-5639) was reported by Sveta Smirnova.
• Bug #95928 – “InnoDB crashes on compressed and encrypted table when changing FS block size“. Make sure to check and write original filesystem block size somewhere… This bug was reported by Sergei Glushchenko from Percona.
• Bug #95929 – “Duplicate entry for key ‘PRIMARY’ when querying information_schema.TABLES”. This nice feature of new data dictionary of MySQL 8.0.16 under concurrent load was found by Manuel Rigger.
• Bug #95934 – “innodb_flush_log_at_trx_commit = 0 get performance regression on 8 core machine“. If I had access to 8 cores box, I’d try to check this sysbench-based complete test case immediately. I should do this on my quadcore box at least, next week. But here we see some 12 days just wasted and then bug report from Chen Zongzh ended up “Analyzing”, till today, with no further feedback.
• Bug #95954 – “CAST of negative function return value to UNSIGNED malfunctions with BIGINT“. Yet another weird bug found by Manuel Rigger. There are patches (separate for 5.6, 5.7 and 8.0) contributed by Oleksandr Peresypkin.

Montjuïc is a place to enjoy nice view of Barcelona and forget about MySQL bugs. So, I could miss some interesting ones at the end of May, 2019…

To summarize:

1. Percona engineers still contribute a lot of useful and interesting MySQL bug reports, often – with patches.
2. I am impressed by quality of MySQL bug reports in 2019. Often we see detailed analysis, proper complete test cases and patches.
3. There is a recent fork of DBD::mysql for MariaDB (and MySQL), DBD::MariaDB!
4. I badly need some new hardware for running benchmarks and tests from recent bug reports…
5. I should spend more time in May in Barcelona

Feed: Planet PostgreSQL.

Built-in Sharding Architecture

The build-in sharding feature in PostgreSQL is using the FDW based approach, the FDW’s are based on sql/med specification that defines how an external data source can be accessed from the PostgreSQL server. PostgreSQL provides number of foreign data wrapper (FDW’s) that are used for accessing external data sources, the postgres_fdw is used for accessing Postgres database running on external server, MySQL_fdw is used for accessing MySQL database from PG, MongoDB_fdw is used for accessing MongoDB and so on.

The diagram below explains the current approach of built-in Sharding in PostgreSQL, the partitions are created on foreign servers and PostgreSQL FDW is used for accessing the foreign servers and using the partition pruning logic the planner decides which partition to access and which partitions to exclude from the search.

Push Down Capabilities

Push down in this context is the ability to push parts of the foreign query to foreign servers in order to decrease the amount of data travelling from the foreign server to parent node. The two basic push-down techniques that have been part of postgres fdw from the start are select target-list pushdown and WHERE clause pushdown.

In the query above the planner will decide which partition to access based on the partition key i.e. logdate in this case. The WHERE clause will be pushed down to the foreign server that contains the respective partition. That’s the basic push down capabilities available in postgres_fdw.

The sharding feature requires more advanced push-down capabilities in order to push the maximum operations down to the foreign servers containing partitions and minimising the data sent over the wire to the parent node.

The above is a decent set of push down capabilities that have been added to PostgreSQL in last few major releases. The good thing about these features is that it already benefits a number of use cases even when the entire sharding feature is not in place.

What’s remaining and associated challenges

The are still a number of important features remaining before we can say that we have Sharding feature in PostgreSQL. In this section we are going to discuss these features and what are the challenges with these features. I am sure there are other features related to database cluster management i.e. backup/failover or monitoring that are not in this list.

1- 2PC for foreign data wrapper transactions

Currently FDW transactions don’t
support two phase commit, this means that if you are using multiple foreign
servers in a transaction and if one part of transaction fails in one foreign
server then the entire transaction on all foreign serves are suppose to fail.
This feature is required in order to guarantee data consistency across the
database cluster.

This feature is required in order to
support OLTP workload hence it is very important for sharding feature.

The design proposal and patches for this feature has been sent on hackers for last several years but it is not getting enough community interest hence the design of this feature is still outstanding.

2- Parallel foreign scan

When a query is querying multiple foreign scans in a single query, all the foreign scans are being executed in a sequential manner, one after another. The parallel foreign scan functionality is executing multiple foreign scans in parallel. This feature is really important for the OLAP test cases, for example if you are running AVG query on a large partition table that is divided over large number of partitions. The AVG operation will be sent to each foreign server sequentially and results from each foreign server is sent to the parent node which will be aggregate on the parent node and sent back to client. Once we have the parallel foriegn scan functionality, all the average operations on all the foreign servers will be executed in parallel and results sent to the parent node. The parent node will aggregate the data and sent the results to the client.

This is key piece needed for
completing the sharding feature, we currently have aggregate pushdown that will
send the aggregates down to the foreign server but we don’t have the
functionality to run the aggregate operations on all the partitions in
parallel.

This feature is particularly very
important for the OLAP use-case, the idea of having a large number of foreign
servers containing partition for a large partitioned table and aggregate
operation on partition running on all the foreign servers in parallel is very
powerful.

The infrastructure for parallel foreign scan feature is asynchronous query execution, this is a major change in PostgreSQL. There has been some work done on this but it feels that it is still a release or two away before it will be committed. Once asynchronous query execution is done, it will be easier to add parallel foreign scan functionality.

3- Shard management

The partitions on foreign servers are currently not getting created automatically, as described in “Sharding in PostgreSQL” section, the partitions needs to be created manually on foreign servers. This can be very tedious task if you are creating a partition table with large number of partitions and sub-partitions.

The shard management feature is
suppose to provide the ability to auto-create the partitions and sub-partitions
on the foreign servers. This will make the creation of sharded tables very
easy.

Not intending to go into any design details of how this feature will be implemented, the basic idea is that Sharded table syntax will be built on top on declarative sharding syntax. The postgres_fdw will be used to pushdown the DDL to the foreign servers, while the FDW’s are only meant to do SELECT or DML, doing DDL on external source is not part of sql/med specification. Anyhow we aren’t suppose to discuss the design of this feature in this blog.

This feature is not yet started in the community, the development team at HighGo is planning to work on this feature.

4- Global Transaction Manager / Snapshot Manager

This is another very important and
difficult feature that is mandatory for Sharding feature. The purpose of global
transaction/snapshot manager is suppose to provide global transactional
consistency. The problem described in section 1 of this chapter “2PC for
foreign data wrapper transactions” also ties in with the Global transaction
manager.

Lets suppose you have two concurrent
clients that are using a sharded table, client #1 is trying to access a
partition that is on server 1 and client #2 is also trying to access the
partition that is also on server 1. Client 2 should get a consistent view of
the partition i.e. any changes i.e. updates etc made to the partition during
client 1 transaction shouldn’t be visible to client 2. Once client 1
transaction gets committed, the charges will be visible to all new
transactions. The Global transaction manager is suppose to ensure that all
global transaction gets a consistent view of the database cluster. All the
concurrent clients using the database cluster (with tables sharded across
multiple foreign servers) should see consistent view of the database cluster.

This is hard problem to solve and companies like Postgres Professional have tried to solve this problem by using a external transaction manager. So far there doesn’t seem to be any solution accepted by the community. Right now there is no visible concentrated effort which is trying to implement the global transaction manager in the core or even as an external component.

There is mention of using other approaches like Clock-SI (Snapshot isolation for Partitioned tables) approach that is followed by other successful projects like Google cloud spanner and YugaByte for solving the same problem.

Conclusion

This is conclusion of all the 3 blogs of this series, horizontal scalability with sharding is imperative for PostgreSQL. It is possible that only some of the workloads need sharding today in order to solve there problems but I am sure everyone wants to know that PostgreSQL has a answer of this problem. It is also important to note that Sharding is not a solution for all big data or high concurrent workloads, you need to pick workloads where larger table can be logically partitioned across partitions and the queries are benefiting from the pushdown and other capabilities in using the sharded cluster.

As I mentioned in the initial section of this blog, the first target for the sharding features where it is complete is to be able to speed-up a long running complex query. This would be a OLAP query, not to say that sharing would benefit the OLTP workloads. The data would be partitioned across multiple servers instead of a single server.

Another important exercise that the sharding team should start to do soon is benchmarking using the capabilities already part of PostgreSQL. I know without parallel foreign scan, it is not possible to speed up a real OLAP query that uses multiple partitions. However the process of benchmarking should being soon, we need to identify the type of workload that should benefit from sharding, what is the performance without sharding and what performance to expect with a sharded cluster. I don’t think we can expect the performance to be linear as we add more shards to the cluster.

Another important point that I would to mention here that there has been critics about using the FDW machinery for implementing built-in sharding. There has been suggestion to go a more low level in order to efficiently handle cross node communication etc. The answer given by a senior community member is good one, we are using FDW machinery to implement this feature because that’s the quickest and less error prone route for implementing it.  The FDW functionality is already tried and tested, if we try to implement using a approach that’s more complex and sophisticated,  it will require allot of recourses and lots of time before we can produce something that we can call sharding.

It will take more then a few companies to invest there resources in building this big feature. So more and more companies should come together on implementing this feature in the community because it is Worth it.

Ahsan Hadi is a VP of Development with HighGo Software Inc. Prior to coming to HighGo Software, Ahsan had worked at EnterpriseDB as a Senior Director of Product Development, Ahsan worked with EnterpriseDB for 15 years. The flagship product of EnterpriseDB is Postgres Plus Advanced server which is based on Open source PostgreSQL. Ahsan has vast experience with Postgres and has lead the development team at EnterpriseDB for building the core compatibility of adding Oracle compatible layer to EDB’s Postgres Plus Advanced Server. Ahsan has also spent number of years working with development team for adding Horizontal scalability and sharding to Postgres. Initially, he worked with postgres-xc which is multi-master sharded cluster and later worked on managing the development of adding horizontal scalability/sharding to Postgres. Ahsan has also worked a great deal with Postgres foreign data wrapper technology and worked on developing and maintaining FDW’s for several sql and nosql databases like MongoDB, Hadoop and MySQL.

Prior to EnterpriseDB, Ahsan worked for Fusion Technologies as a Senior Project Manager. Fusion Tech was a US based consultancy company, Ahsan lead the team that developed java based job factory responsible for placing items on shelfs at big stores like Walmart. Prior to Fusion technologies, Ahsan worked at British Telecom as a Analyst/Programmer and developed web based database application for network fault monitoring.

Ahsan joined HighGo Software Inc (Canada) in April 2019 and is leading the development teams based in multiple Geo’s, the primary responsibility is community based Postgres development and also developing HighGo Postgres server.

Feed: Planet PostgreSQL.

Replication is a critical part of any database system that aims to provide high availability (HA) and effective disaster recovery (DR) strategy. This blog is aimed at establishing the role of replication in a database system. The blog will give a general overview of replication and its types as well as an introduction to replication options in PostgreSQL.

The term Replication is used to describe the process of sharing information between one or more software or hardware systems; to ensure reliability, availability, and fault-tolerance. These systems can be located in the same vicinity, could be on a single machine or perhaps connected over a wide network. The replication can be divided broadly into Hardware and Software categories. We’ll explore these categories briefly, however, the main focus of this blog is database replication. So, let’s first understand what constitutes a database replication system.

In a nutshell, the database replication describes the process of coping data from the database instance to one or more database instances. Again, these instances can be in the same location or perhaps connected over a wide network.

Hardware-Based Replication

Let’s start with the hardware replication. Hardware-based replication keeps the multiple connected systems in sync. This syncing of data is done at the storage level as soon as an I/O is performed on the system, it’s propagated to the configured storage modules/devices/systems. This type of replication can be done for the entire storage or the selected partitions. The biggest advantage of this type of solution is that (generally) it’s easier to set up and is independent of software. This makes it perform much better, however, it reduces the flexibility and control over the replication. Here are the few pros of this type of replication.

Real-time – all the changes are applied to subsequent systems immediately.
Easier to Setup – no scripting or software configurations are required.
Independent of Application – replication happens at the storage layer and is independent of OS/software application.
Data Integrity and Consistency – As the mirroring happens at the storage layer, which effectively makes an exact copy of the storage disk, So the data integrity and consistency are automatically ensured.

Although hardware replication has some very appealing advantages yet it comes with its own limitations. It generally relies on the vendor locking i.e. the same type of hardware has to be used and often it’s not very cost-effective.

Software-Based Replication

This replication can range from generic solutions to product-specific solutions. The generic solutions tend to emulate the hardware replication by copying the data between different system at the software level. The software that is responsible for performing the replication, copies each bit written to the source storage and propagates it to the destination system(s). Whereas the product-specific solutions, are more considerate towards the product requirements and are generally meant for a specific product. The software-based replication has its pros and cons. On one hand, It provides flexibility and control over how the replication is done and is usually very cost-effective while provides a much better set of features. But on the other hand, it requires a lot of configurations and requires continuous monitoring and maintenance.

Database Replication

Having discussed replication and its different types; lets now turn our focus toward the database replication topic.

Before going into more details lets start by discussing the different terminologies used to describe the different components of a replication system in the database world. Primary-Standby, Master-Slave, Publisher-Subscriber, Master-Master/Multimaster are the most often used terms to describe the database servers participating in the replication setup.

The term Primary, Master and Publisher are used to describing the active node that strives to propagate changes received by it to the other nodes. Whereas Standby, Slave and Subscriber terms are used to describe the passive nodes that will be receiving the propagated changes from the active nodes. In this blog, we will use Primary to describe the active node and Standby for the passive node.

The database replication can be configured in Primary-Standby and Multi-master configurations.

In a Primary-Standby configuration, only one instance receives the data changes and then it distributes them among the standbys.

Whereas in the Multi-master system, each instance of a database can receive the data change and then propagates that change to other instances.

Synchronous/Asynchronous Replication

At the core of the replication process is the ability of the primary node to transmit data to standbys. Much like any other data transfer strategy, this can happen in a synchronous way where primary waits for all standbys to confirm that they have received and written the data on disk. The client is given a confirmation when these acknowledgments are received. Alternatively, the primary node can commit the data locally and transmit the transaction data to standbys whenever possible with the expectation that standbys will receive and write the data on disk. In this case, standbys do not send confirmation to the primary. Whereas the former strategy is called synchronous replication, this is referred to as asynchronous replication. Both hardware and software-based solutions support synchronous and asynchronous replication.

Following are diagrams that show both types of replication strategies in a graphical way.

Since the main difference between the two strategies is the acknowledgment of data written on the standby systems; there are advantages and disadvantages in using both techniques. Synchronous replication may be configured in a way such that all systems are up-to-date and that the replication is done in real-time. But at the same time, it adversely impacts the performance of the system since the primary node waits for standbys’ confirmations.

The asynchronous replication tends to give better performance as there is no wait time attributed to confirmation messages from standbys.

Cascading Replication

So far we have seen that only the primary node transmits the transaction data to standbys. This is a load that can be distributed among multiple systems; i.e. a standby that has received the data and written on disk can transmit it onwards to standbys that are configured to receive data from it. This is called cascading replication where replication is configured between primary and standbys, and standby(s) and standbys. Following is a visual representation of cascading replication.

Standby Modes

Warm standby is a term used to describe a standby server that allows standby to receive the changes from primary but does not allow any of the client connection to it directly.

Hot standby is a term used to describe a standby server which also allows accepting the client connections.

PostgreSQL Replication

In the end, I would like to share the replication options available for PostgreSQL.

PostgreSQL offers built-in log streaming and logical replication options. The built-in replication is only available in the primary-standby configuration.

Streaming Replication (SR)

Also known as physical and binary replication in Postgres. Binary data is streamed to the standbys. In order to stream such data, streaming replication uses the WAL (write-ahead log) records. Any change made on the primary server is first written to the WAL files before it is written on disk and database server has the capability to stream these records to any number of standbys. Since its a binary copy of data, there are certain limitations to it as well. This type of replication can only be used when all of the changes have to be replicated on the standbys. One cannot use this if only a subset of changes is required. The standby servers either do not accept the connections or if they do, they can only serve the read-only queries. The other limitation is that it can not be used with a different version of the database server. The whole setup, consisting of multiple database servers has to be of the same version.

Streaming replication is asynchronous by default, however, it not much difficult to turn on the synchronous replication. This kind of setup is perfect for achieving high availability (HA). If the main server fails, one of the standbys can take its place since they are almost the exact copy of it. Here is the basic configuration:

# First create a user with replication role in the primary database, this user will be used by standbys to establish a connection with the primary:
CREATE USER foouser REPLICATION;

# Add it to the pg_hba.conf to allow authentication for this user:
# TYPE  DATABASE        USER       ADDRESS                 METHOD
host    replication     foouser          trust

# Take a base backup on the stanby:
pg_basebackup -h  -U foouser -D ~/standby

# In postgresql.conf following entries are needed
wal_level=replica # should be set to replica for streaming replication
max_wal_senders = 8 # number of standbys to allow connection at a time

# On the stanby, create a recovery.conf file in its data direcrtory with following contents:
standby_mode=on
primary_conninfo='user=foouser host= port='

Now start the servers and a basic streaming replication should work.

Logical Replication

Logical Replication is considerably new in PostgreSQL as compared to the streaming replication. The streaming replication is a byte-by-byte copy of the entire data. It, however, does not allow replication of a single table or a subset of the data from a primary server to standbys. Logical replication enables replicating particular objects in the database to the standbys, instead of replicating the whole database. It also allows replicating data between different versions of the database servers and not only that it also allows the standby to accept both read and write queries. However, be watchful of the fact that there is no conflict resolution system implemented. So if both primary and standbys are allowed to write on the same table, more likely than not, there will be data conflicts that will stop the replication process. In this case, the user must manually resolve these conflicts. However, both standbys and primary may allow writing on complete disjointed sets of data which will avoid any conflict resolution needs. Here is the basic configuration:

# First create a user with replication role in the primary database, this user will be used by standbys to establish a connection with the primary:
CREATE USER foouser REPLICATION;

# Add it to the pg_hba.conf to allow authentication for this user:
# TYPE  DATABASE        USER       ADDRESS                 METHOD
host    replication     foouser          trust

# For logical replication, base backup is not required, any of the two database instances can be used to create this setup.

wal_level=logical # for logical replication, wal_level needs to be set to logical in postgresql.conf file.

# on primary server, create a some tables to replicate and a publication that will list down these tables.
CREATE TABLE t1 (col1 int, col2 varchar);
CREATE TABLE t2 (col1 int, col2 varchar);
INSERT INTO t1 ....;
INSERT INTO t2 ....;
CREATE PUBLICATION foopub FOR TABLE t1, t2;

# on the standby, the structure of above mentioned tables needs to be created as DDL is not replicated.
CREATE TABLE t1 (col1 int, col2 varchar);
CREATE TABLE t2 (col1 int, col2 varchar);

# on standby, create a subscription for the above publication.
CREATE SUBSCRIPTION foosub CONNECTION 'host= port= user=foouser' PUBLICATION foopub;

With this a basic logical replication is achieved.

I hope this blog helped you understand replication in general and from PostgreSQL perspective streaming and logical replications concepts. Furthermore, I hope this helps you in better designing a replication environment for your needs.

Asif Rehman is a Senior Software Engineer at HighGo Software. He Joined EnterpriseDB, an Enterprise PostgreSQL’s company in 2005 and started his career in open source development particularly in PostgreSQL. Asif’s contributions range from developing in-house features relating to oracle compatibility, to developing tools around PostgreSQL. He Joined HighGo Software in the month of Sep 2018.

Feed: Planet MySQL
;
Author: Dave Stokes
;
Earlier I wrote about check constraints when MySQL 8.0.16 was released. But this week I noticed two different folks having similar problems with them. And sadly it is ‘pilot error’.

The first was labeled  MYSQL 8.0.17 CHECK not working even though it has been implemented and a cursory glance may make one wonder what is going on with the database.

The table is set up with two constraints. And old timers will probably mutter something under their breath about using ENUMs but here they are:

 JOB_TITLE varchar(20) CHECK(JOB_TITLE IN (‘Lecturer’, ‘Professor’, ‘Asst. Professor’, ‘Sr. Lecturer’)),  

Feed: Planet MySQL
;
Author: Federico Razzoli
;

Transactions are the least known RDBMS features. Everyone knows they exist, but few know how they work. For this reason, it is not obvious that they can be read only. This article explains what read only transactions are and why we should use them when appropriate.

What is a read only transaction?

An objection I’ve heard more than once is: transactions are for writes, SELECTs don’t even happen inside a transaction! But this is wrong. Transactions are not only about ROLLBACK and COMMIT. They are also about isolation.

As a general rule, this is what happens. When you start a transaction we acquire a view, or snapshot, on data. Until the end of the transaction we will see that snapshot, not modifications made by other connections.

Reads are generally non-locking. After you’ve read a row, other users can still modify it, even if your transaction is still in progress. Internally, MySQL will keep the information that allow to “rebuild” data as they were when you acquired the snapshot, so you can still see them. This information is preserved in the transaction logs.

To be clear, we actually always use transactions. By default, we run queries in autocommit mode, and we don’t start or end transactions explicitly. But even in this case, every SQL statement we run is a transaction.

There is no such thing as reading objective data. We always read from (or write to) snapshots acquired at a certain point of time, that include all the changes made by committed transactions.

Why should we use read only transactions?

The answer should be obvious now. When we want to run more than one SELECT, and we want the results to be consistent, we should do it in a read only transaction. For example, we could do something like this:

SELECT COUNT(DISTINCT user_id)
    FROM product WHERE date = DATE();
SELECT COUNT(DISTINCT product_id)
    FROM product WHERE date = DATE();

If we want the two results to reflect the reality at a certain point in time, we need to do this in a transaction:

START TRANSACTION READ ONLY;
SELECT COUNT(DISTINCT user_id)
    FROM product WHERE date = DATE();
SELECT COUNT(DISTINCT product_id)
    FROM product WHERE date = DATE();
COMMIT;

The question implies something: why should we use a read only transaction, when we can use a regular read write transaction? If for some reason we try to write something, the statement will fail. So it’s simpler to just always use read write transactions.

The reason for using read only transactions is mentioned in the MySQL manual:

InnoDB can avoid the overhead associated with setting up the transaction ID (TRX_ID field) for transactions that are known to be read-only. A transaction ID is only needed for a transaction that might perform write operations or locking reads such as SELECT ... FOR UPDATE. Eliminating unnecessary transaction IDs reduces the size of internal data structures that are consulted each time a query or data change statement constructs a read view.

8.5.3 Optimizing InnoDB Read-Only Transactions – 25 August 2019

Isolation levels

Isolation level is a setting that defines how much the current transaction is isolated from concurrent transactions. It can be changed at session level, before starting the next transaction. There is also a global setting, which takes effect for sessions that don’t change their isolation level. By default it is REPEATABLE READ.

Isolation levels are:

• READ UNCOMMITTED;
• READ COMMITTED;
• REPEATABLE READ.

(We’re skipping SERIALIZABLE here, as it’s just a variation of REPEATABLE READ and it does not make sense with read only transactions)

This list is only valid for MySQL. Different DBMSs support different isolation levels.

The behavior described above refers to REPEATABLE READ. To recap, it acquires a snapshot of the data, which is used for the whole duration of the transaction.

Another isolation level is READ COMMITTED. It acquires a new snapshot for each query in the transaction. But this does not make sense for a read only transaction.

A usually underestimated isolation level is READ UNCOMMITTED. The reason why it is underestimated is that it reads data that are being written by other transactions, but are not yet committed – and possibly, will never be. For this reason, highly inconsistent data may be returned.

However, this can be an important optimisation. In my experience, it can make some queries much faster and greatly reduce CPU usage, especially when you are running expensive analytics SELECTs on write-intensive servers.

There are cases when results inconsistencies don’t matter. A typical example is running aggregations on many rows: an average on one million values will not change sensibly. Another example is when reading session-level data, that no one else is modifying; or old data, that no one is supposed to modify anymore.

If you want to use READ UNCOMMITTED, there is no reason to wrap multiple queries in a single transaction. For each query, you can run something like this:

SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
START TRANSACTION;
SELECT COUNT(*) FROM product;
COMMIT;

MetaData Lock and the rollback trick

When accessing a table using a non-transactional engine (like MyISAM) inside a transaction, MySQL will return a warning. They seem to assume that there is no reason to access such tables from a transaction, which is false. MariaDB doesn’t return such warning.

In reality, transactions acquire metadata locks (MDL). This means that other connections will not be able to run an ALTER TABLE on those tables until the transaction finishes. Preventing such problems is very important and should be done every time it’s useful.

mysqldump rolls back

mysqldump --single-transaction uses default read write transactions, not read only ones. My guess is that they don’t consider it important for a single long transaction, but only for many concurrent transactions.

However, the MariaDB version of mysqldump does something interesting with rollbacks:

ROLLBACK TO SAVEPOINT in --single-transaction mode to release metadata lock on table which was already dumped. This allows to avoid blocking concurrent DDL on this table without sacrificing correctness, as we won't access table second time and dumps created by --single-transaction mode have validity point at the start of transaction anyway.

Note that this doesn't make --single-transaction mode with concurrent DDL safe in general case. It just improves situation for people for whom it might be working.

mysqldump in MariaDB repo, taken from commit 624dd71b9419555eca8baadc695e3376de72286f

In other words:

• An MDL blocks some operations, so it can be a problem just like any other lock, especially in a long transaction;
• An MDL is only useful for data that we are going to access again, just like any other lock;
• Transactions support savepoints in the middle of a transaction, and we can rollback to a savepoint instead of rolling the whole transaction back;
• Doing so releases the MDLs on objects that were only accessed after the savepoint.

See also

Related courses:

Conclusions

Here I wanted to advertise a feature that, in my opinion, should be used much more often. Actually, if you know from the beginning that a transaction is read only, there is no reason not to inform MySQL about that.

We discussed which isolation levels make sense for read only transactions, and the nice ROLLBACK TO SAVEPOINT trick used by mysqldump for MariaDB.

Did you notice any mistake? Do you have ideas to contribute?
Please comment!

As usual, I’ll be happy to fix errors and discuss your ideas.

Toodle pip,
Federico

Photo source: Wikipedia

Feed: AWS Big Data Blog.

This post describes how you can improve performance and reduce costs by offloading reporting workloads from an online transaction processing (OLTP) database to Amazon Athena and Amazon S3. The architecture described allows you to implement a reporting system and have an understanding of the data that you receive by being able to query it on arrival. In this solution:

• Oracle GoldenGate generates a new row on the target for every change on the source to create Slowly Changing Dimension Type 2 (SCD Type 2) data.
• Athena allows you to run ad hoc queries on the SCD Type 2 data.

Principles of a modern reporting solution

Advanced database solutions use a set of principles to help them build cost-effective reporting solutions. Some of these principles are:

• Separate the reporting activity from the OLTP. This approach provides resource isolation and enables databases to scale for their respective workloads.
• Use query engines running on top of distributed file systems like Hadoop Distributed File System (HDFS) and cloud object stores, such as Amazon S3. The advent of query engines that can run on top of open-source HDFS and cloud object stores further reduces the cost of implementing dedicated reporting systems.

Furthermore, you can use these principles when building reporting solutions:

• To reduce licensing costs of the commercial databases, move the reporting activity to an open-source database.
• Use a log-based, real-time, change data capture (CDC), data-integration solution, which can replicate OLTP data from source systems, preferably in real-time mode, and provide a current view of the data. You can enable the data replication between the source and the target reporting systems using database CDC solutions. The transaction log-based CDC solutions capture database changes noninvasively from the source database and replicate them to the target datastore or file systems.

Prerequisites

If you use GoldenGate with Kafka and are considering cloud migration, you can benefit from this post. This post also assumes prior knowledge of GoldenGate and does not detail steps to install and configure GoldenGate. Knowledge of Java and Maven is also assumed. Ensure that a VPC with three subnets is available for manual deployment.

Understanding the architecture of this solution

The following workflow diagram (Figure 1) illustrates the solution that this post describes:

1. Amazon RDS for Oracle acts as the source.
2. A GoldenGate CDC solution produces data for Amazon Managed Streaming for Apache Kafka (Amazon MSK). GoldenGate streams the database CDC data to the consumer. Kafka topics with an MSK cluster receives the data from GoldenGate.
3. The Apache Flink application running on Amazon EMR consumes the data and sinks it into an S3 bucket.
4. Athena analyzes the data through queries. You can optionally run queries from Amazon Redshift Spectrum.

Data Pipeline

Figure 1

Amazon MSK is a fully managed service for Apache Kafka that makes it easy to provision  Kafka clusters with few clicks without the need to provision servers, storage and configuring Apache Zookeeper manually. Kafka is an open-source platform for building real-time streaming data pipelines and applications.

Amazon RDS for Oracle is a fully managed database that frees up your time to focus on application development. It manages time-consuming database administration tasks, including provisioning, backups, software patching, monitoring, and hardware scaling.

GoldenGate is a real-time, log-based, heterogeneous database CDC solution. GoldenGate supports data replication from any supported database to various target databases or big data platforms like Kafka. GoldenGate’s ability to write the transactional data captured from the source in different formats, including delimited text, JSON, and Avro, enables seamless integration with a variety of BI tools. Each row has additional metadata columns including database operation type (Insert/Update/Delete).

Flink is an open-source, stream-processing framework with a distributed streaming dataflow engine for stateful computations over unbounded and bounded data streams. EMR supports Flink, letting you create managed clusters from the AWS Management Console. Flink also supports exactly-once semantics with the checkpointing feature, which is vital to ensure data accuracy when processing database CDC data. You can also use Flink to transform the streaming data row by row or in batches using windowing capabilities.

S3 is an object storage service with high scalability, data availability, security, and performance. You can run big data analytics across your S3 objects with AWS query-in-place services like Athena.

Athena is a serverless query service that makes it easy to query and analyze data in S3. With Athena and S3 as a data source, you define the schema and start querying using standard SQL. There’s no need for complex ETL jobs to prepare your data for analysis, which makes it easy for anyone familiar with SQL skills to analyze large-scale datasets quickly.

The following diagram shows a more detailed view of the data pipeline:

1. RDS for Oracle runs in a Single-AZ.
2. GoldenGate runs on an Amazon EC2 instance.
3. The MSK cluster spans across three Availability Zones.
4. Kafka topic is set up in MSK.
5. Flink runs on an EMR Cluster.
6. Producer Security Group for Oracle DB and GoldenGate instance.
7. Consumer Security Group for EMR with Flink.
8. Gateway endpoint for S3 private access.
9. NAT Gateway to download software components on GoldenGate instance.
10. S3 bucket and Athena.

For simplicity, this setup uses a single VPC with multiple subnets to deploy resources.

Figure 2

Configuring single-click deployment using AWS CloudFormation

The AWS CloudFormation template included in this post automates the deployment of the end-to-end solution that this blog post describes. The template provisions all required resources including RDS for Oracle, MSK, EMR, S3 bucket, and also adds an EMR step with a JAR file to consume messages from Kafka topic on MSK. Here’s the list of steps to launch the template and test the solution:

1. Launch the AWS CloudFormation template in the us-east-1
2. After successful stack creation, obtain GoldenGate Hub Server public IP from the Outputs tab of cloudformation.
3. Login to GoldenGate hub server using the IP address from step 2 as ec2-user and then switch to oracle user.sudo su – oracle
4. Connect to the source RDS for Oracle database using the sqlplus client and provide password(source).[oracle@ip-10-0-1-170 ~]$ sqlplus source@prod
5. Generate database transactions using SQL statements available in oracle user’s home directory.

   SQL> @s
   
    SQL> @s1
   
    SQL> @s2

6. Query STOCK_TRADES table from Amazon Athena console. It takes a few seconds after committing transactions on the source database for database changes to be available for Athena for querying.

Manually deploying components

The following steps describe the configurations required to stream Oracle-changed data to MSK and sink it to an S3 bucket using Flink running on EMR. You can then query the S3 bucket using Athena. If you deployed the solution using AWS CloudFormation as described in the previous step, skip to the Testing the solution section.

1. Prepare an RDS source database for CDC using GoldenGate.The RDS source database version is Enterprise Edition 12.1.0.2.14. For instructions on configuring the RDS database, see Using Oracle GoldenGate with Amazon RDS. This post does not consider capturing data definition language (DDL).
2. Configure an EC2 instance for the GoldenGate hub server.Configure the GoldenGate hub server using Oracle Linux server 7.6 (ami-b9c38ad3) image in the us-east-1 Region. The GoldenGate hub server runs the GoldenGate extract process that extracts changes in real time from the database transaction log files. The server also runs a replicat process that publishes database changes to MSK.The GoldenGate hub server requires the following software components:

• Java JDK 1.8.0 (required for GoldenGate big data adapter).
• GoldenGate for Oracle (12.3.0.1.4) and GoldenGate for big data adapter (12.3.0.1).
• Kafka 1.1.1 binaries (required for GoldenGate big data adapter classpath).
• An IAM role attached to the GoldenGate hub server to allow access to the MSK cluster for GoldenGate processes running on the hub server.Use the GoldenGate (12.3.0) documentation to install and configure the GoldenGate for Oracle database. The GoldenGate Integrated Extract parameter file is eora2msk.prm.

  EXTRACT eora2msk
  SETENV (NLSLANG=AL32UTF8)
  
  USERID ggadmin@ORCL, password ggadmin
  TRANLOGOPTIONS INTEGRATEDPARAMS (max_sga_size 256)
  EXTTRAIL /u01/app/oracle/product/ogg/dirdat/or
  LOGALLSUPCOLS
  
  TABLE SOURCE.STOCK_TRADES;

  The logallsupcols extract parameter ensures that a full database table row is generated for every DML operation on the source, including updates and deletes.

3. Create a Kafka cluster using MSK and configure Kakfa topic.You can create the MSK cluster from the AWS Management Console, using the AWS CLI, or through an AWS CloudFormation template.

• Use the list-clusters command to obtain a ClusterArn and a Zookeeper connection string after creating the cluster. You need this information to configure the GoldenGate big data adapter and Flink consumer. The following code illustrates the commands to run:

  $aws kafka list-clusters --region us-east-1
  {
      "ClusterInfoList": [
          {
              "EncryptionInfo": {
                  "EncryptionAtRest": {
                      "DataVolumeKMSKeyId": "arn:aws:kms:us-east-1:xxxxxxxxxxxx:key/717d53d8-9d08-4bbb-832e-de97fadcaf00"
                  }
              }, 
              "BrokerNodeGroupInfo": {
                  "BrokerAZDistribution": "DEFAULT", 
                  "ClientSubnets": [
                      "subnet-098210ac85a046999", 
                      "subnet-0c4b5ee5ff5ef70f2", 
                      "subnet-076c99d28d4ee87b4"
                  ], 
                  "StorageInfo": {
                      "EbsStorageInfo": {
                          "VolumeSize": 1000
                      }
                  }, 
                  "InstanceType": "kafka.m5.large"
              }, 
              "ClusterName": "mskcluster", 
              "CurrentBrokerSoftwareInfo": {
                  "KafkaVersion": "1.1.1"
              }, 
              "CreationTime": "2019-01-24T04:41:56.493Z", 
              "NumberOfBrokerNodes": 3, 
              "ZookeeperConnectString": "10.0.2.9:2181,10.0.0.4:2181,10.0.3.14:2181", 
              "State": "ACTIVE", 
              "CurrentVersion": "K13V1IB3VIYZZH", 
              "ClusterArn": "arn:aws:kafka:us-east-1:xxxxxxxxx:cluster/mskcluster/8920bb38-c227-4bef-9f6c-f5d6b01d2239-3", 
              "EnhancedMonitoring": "DEFAULT"
          }
      ]
  }

• Obtain the IP addresses of the Kafka broker nodes by using the ClusterArn.

  $aws kafka get-bootstrap-brokers --region us-east-1 --cluster-arn arn:aws:kafka:us-east-1:xxxxxxxxxxxx:cluster/mskcluster/8920bb38-c227-4bef-9f6c-f5d6b01d2239-3
  {
      "BootstrapBrokerString": "10.0.3.6:9092,10.0.2.10:9092,10.0.0.5:9092"
  }

• Create a Kafka topic. The solution in this post uses the same name as table name for Kafka topic.

  ./kafka-topics.sh --create --zookeeper 10.0.2.9:2181,10.0.0.4:2181,10.0.3.14:2181 --replication-factor 3 --partitions 1 --topic STOCK_TRADES

4. Provision an EMR cluster with Flink.Create an EMR cluster 5.25 with Flink 1.8.0 (advanced option of the EMR cluster), and enable SSH access to the master node. Create and attach a role to the EMR master node so that Flink consumers can access the Kafka topic in the MSK cluster.
5. Configure the Oracle GoldenGate big data adapter for Kafka on the GoldenGate hub server.Download and install the Oracle GoldenGate big data adapter (12.3.0.1.0) using the Oracle GoldenGate download link. For more information, see the Oracle GoldenGate 12c (12.3.0.1) installation documentation.The following is the GoldenGate producer property file for Kafka (custom_kafka_producer.properties):

   #Bootstrap broker string obtained from Step 3
   bootstrap.servers= 10.0.3.6:9092,10.0.2.10:9092,10.0.0.5:9092
   #bootstrap.servers=localhost:9092
   acks=1
   reconnect.backoff.ms=1000
   value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
   key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
   # 100KB per partition
   batch.size=16384
   linger.ms=0

   The following is the GoldenGate properties file for Kafka (Kafka.props):

   gg.handlerlist = kafkahandler
   gg.handler.kafkahandler.type=kafka
   gg.handler.kafkahandler.KafkaProducerConfigFile=custom_kafka_producer.properties
   #The following resolves the topic name using the short table name
   #gg.handler.kafkahandler.topicName=SOURCE
   gg.handler.kafkahandler.topicMappingTemplate=${tableName}
   #The following selects the message key using the concatenated primary keys
   gg.handler.kafkahandler.keyMappingTemplate=${primaryKeys}
   gg.handler.kafkahandler.format=json_row
   #gg.handler.kafkahandler.format=delimitedtext
   #gg.handler.kafkahandler.SchemaTopicName=mySchemaTopic
   #gg.handler.kafkahandler.SchemaTopicName=oratopic
   gg.handler.kafkahandler.BlockingSend =false
   gg.handler.kafkahandler.includeTokens=false
   gg.handler.kafkahandler.mode=op
   goldengate.userexit.writers=javawriter
   javawriter.stats.display=TRUE
   javawriter.stats.full=TRUE
   
   gg.log=log4j
   #gg.log.level=INFO
   gg.log.level=DEBUG
   gg.report.time=30sec
   gg.classpath=dirprm/:/home/oracle/kafka/kafka_2.11-1.1.1/libs/*
   
   javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar

   The following is the GoldenGate replicat parameter file (rkafka.prm):

   REPLICAT rkafka
   -- Trail file for this example is located in "AdapterExamples/trail" directory
   -- Command to add REPLICAT
   -- add replicat rkafka, exttrail AdapterExamples/trail/tr
   TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka.props
   REPORTCOUNT EVERY 1 MINUTES, RATE
   GROUPTRANSOPS 10000
   MAP SOURCE.STOCK_TRADES, TARGET SOURCE.STOCK_TRADES;

6. Create an S3 bucket and directory with a table name underneath for Flink to store (sink) Oracle CDC data.
7. Configure a Flink consumer to read from the Kafka topic that writes the CDC data to an S3 bucket.For instructions on setting up a Flink project using the Maven archetype, see Flink Project Build Setup.The following code example is the pom.xml file, used with the Maven project. For more information, see Getting Started with Maven.

   
     4.0.0
   
     org.apache.flink
     flink-quickstart-java
     1.8.0
     jar
   
     flink-quickstart-java
     http://www.example.com
   
     
       UTF-8
       @slf4j.version@
       @log4j.version@
       1.8
       1.7
       1.7
     
   
   
       
           org.apache.flink
           flink-java
           1.8.0
           compile
       
       
           org.apache.flink
           flink-hadoop-compatibility_2.11
           1.8.0
       
       
        org.apache.flink
        flink-connector-filesystem_2.11
        1.8.0
       
   
       
           org.apache.flink
           flink-streaming-java_2.11
           1.8.0
           compile
       
        
           org.apache.flink
           flink-s3-fs-presto
           1.8.0
       
       
      org.apache.flink
           flink-connector-kafka_2.11
           1.8.0
       
       
         org.apache.flink
         flink-clients_2.11
         1.8.0
       
       
         org.apache.flink
         flink-scala_2.11
         1.8.0
       
   
       
         org.apache.flink
         flink-streaming-scala_2.11
         1.8.0
       
   
       
         com.typesafe.akka
         akka-actor_2.11
         2.4.20
       
       
          com.typesafe.akka
          akka-protobuf_2.11
          2.4.20
       
   
     
        
            org.apache.maven.plugins
               maven-shade-plugin
               3.2.1
               
                      
                         package
                           
                               shade
                            
                          
                         
                     
   
                            
                              
   
                   
                           
                                                                                    org.apache.flink:*
                           
                      
                
                  
                                                                           
                                                                           flinkconsumer.flinkconsumer
                  
                                                                            
                                                                        reference.conf
                                                                   
                                                           
                   
                         
                                                               org.codehaus.plexus.util
                                                                 org.shaded.plexus.util
                       
                                                                     org.codehaus.plexus.util.xml.Xpp3Dom
                                                                     org.codehaus.plexus.util.xml.pull.*
                                                                 
                                                              
                                                           
                                                           false
                                                   
                                           
                                   
                           
   
                           
                                   org.apache.maven.plugins
                                   maven-jar-plugin
                                   2.5
                                   
                                           
                                                   
                                                           flinkconsumer.flinkconsumer
                                                   
                                           
          
                           
   
                           
                                   org.apache.maven.plugins
                                   maven-compiler-plugin
                                   3.1
                                   
                                           1.7
                                           1.7
                                   
                           
                   
   
   
   
                   
                           build-jar
                           
                                   false
                           
                   
           
   
   
   

   Compile the following Java program using mvn clean install and generate the JAR file:

   package flinkconsumer;
   import org.apache.flink.api.common.typeinfo.TypeInformation;
   import org.apache.flink.api.java.typeutils.TypeExtractor;
   import org.apache.flink.api.java.utils.ParameterTool;
   import org.apache.flink.streaming.api.datastream.DataStream;
   import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
   import org.apache.flink.streaming.api.functions.source.SourceFunction;
   import org.apache.flink.streaming.util.serialization.DeserializationSchema;
   import org.apache.flink.streaming.util.serialization.SerializationSchema;
   import org.apache.flink.streaming.util.serialization.SimpleStringSchema;
   import org.apache.flink.api.common.functions.FlatMapFunction;
   import org.apache.flink.api.common.functions.MapFunction;
   import org.apache.flink.streaming.api.windowing.time.Time;
   import org.apache.flink.util.Collector;
   import org.apache.flink.api.java.tuple.Tuple2;
   import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;
   import org.slf4j.LoggerFactory;
   import org.apache.flink.runtime.state.filesystem.FsStateBackend;
   import akka.actor.ActorSystem;
   import akka.stream.ActorMaterializer;
   import akka.stream.Materializer;
   import com.typesafe.config.Config;
   import org.apache.flink.streaming.connectors.fs.*;
   import org.apache.flink.streaming.api.datastream.*;
   import org.apache.flink.runtime.fs.hdfs.HadoopFileSystem;
   import java.util.stream.Collectors;
   import java.util.Arrays;
   import java.util.ArrayList;
   import java.util.Collections;
   import java.util.List;
   import java.util.Properties;
   import java.util.regex.Pattern;
   import java.io.*;
   import java.net.BindException;
   import java.util.*;
   import java.util.Map.*;
   import java.util.Arrays;
   
   public class flinkconsumer{
   
       public static void main(String[] args) throws Exception {
           // create Streaming execution environment
           StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
           env.setBufferTimeout(1000);
           env.enableCheckpointing(5000);
           Properties properties = new Properties();
           properties.setProperty("bootstrap.servers", "10.0.3.6:9092,10.0.2.10:9092,10.0.0.5:9092");
           properties.setProperty("group.id", "flink");
           properties.setProperty("client.id", "demo1");
   
           DataStream message = env.addSource(new FlinkKafkaConsumer<>("STOCK_TRADES", new SimpleStringSchema(),properties));
           env.enableCheckpointing(60_00);
           env.setStateBackend(new FsStateBackend("hdfs://ip-10-0-3-12.ec2.internal:8020/flink/checkpoints"));
   
           RollingSink sink= new RollingSink("s3://flink-stream-demo/STOCK_TRADES");
          // sink.setBucketer(new DateTimeBucketer("yyyy-MM-dd-HHmm"));
          // The bucket part file size in bytes.
              sink.setBatchSize(400);
            message.map(new MapFunction() {
               private static final long serialVersionUID = -6867736771747690202L;
               @Override
               public String map(String value) throws Exception {
                   //return " Value: " + value;
                   return value;
               }
           }).addSink(sink).setParallelism(1);
           env.execute();
       }
   }

   Log in as a Hadoop user to an EMR master node, start Flink, and execute the JAR file:

   $ /usr/bin/flink run ./flink-quickstart-java-1.7.0.jar

8. Create the stock_trades table from the Athena console. Each JSON document must be on a new line.

   CREATE EXTERNAL TABLE `stock_trades`(
     `trade_id` string COMMENT 'from deserializer', 
     `ticker_symbol` string COMMENT 'from deserializer', 
     `units` int COMMENT 'from deserializer', 
     `unit_price` float COMMENT 'from deserializer', 
     `trade_date` timestamp COMMENT 'from deserializer', 
     `op_type` string COMMENT 'from deserializer')
   ROW FORMAT SERDE 
     'org.openx.data.jsonserde.JsonSerDe' 
   STORED AS INPUTFORMAT 
     'org.apache.hadoop.mapred.TextInputFormat' 
   OUTPUTFORMAT 
     'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
   LOCATION
     's3://flink-cdc-demo/STOCK_TRADES'
   TBLPROPERTIES (
     'has_encrypted_data'='false', 
     'transient_lastDdlTime'='1561051196')

   For more information, see Hive JSON SerDe.

Testing the solution

To test that the solution works, complete the following steps:

1. Log in to the source RDS instance from the GoldenGate hub server and perform insert, update, and delete operations on the stock_trades table:

   $sqlplus source@prod
   SQL> insert into stock_trades values(6,'NEW',29,75,sysdate);
   SQL> update stock_trades set units=999 where trade_id=6;
   SQL> insert into stock_trades values(7,'TEST',30,80,SYSDATE);
   SQL>insert into stock_trades values (8,'XYZC', 20, 1800,sysdate);
   SQL> commit;

2. Monitor the GoldenGate capture from the source database using the following stats command:

   [oracle@ip-10-0-1-170 12.3.0]$ pwd
   /u02/app/oracle/product/ogg/12.3.0
   [oracle@ip-10-0-1-170 12.3.0]$ ./ggsci
   
   Oracle GoldenGate Command Interpreter for Oracle
   Version 12.3.0.1.4 OGGCORE_12.3.0.1.0_PLATFORMS_180415.0359_FBO
   Linux, x64, 64bit (optimized), Oracle 12c on Apr 16 2018 00:53:30
   Operating system character set identified as UTF-8.
   
   Copyright (C) 1995, 2018, Oracle and/or its affiliates. All rights reserved.
   
   
   
   GGSCI (ip-10-0-1-170.ec2.internal) 1> stats eora2msk

3. Monitor the GoldenGate replicat to a Kafka topic with the following:

   [oracle@ip-10-0-1-170 12.3.0]$ pwd
   /u03/app/oracle/product/ogg/bdata/12.3.0
   [oracle@ip-10-0-1-170 12.3.0]$ ./ggsci
   
   Oracle GoldenGate for Big Data
   Version 12.3.2.1.1 (Build 005)
   
   Oracle GoldenGate Command Interpreter
   Version 12.3.0.1.2 OGGCORE_OGGADP.12.3.0.1.2_PLATFORMS_180712.2305
   Linux, x64, 64bit (optimized), Generic on Jul 13 2018 00:46:09
   Operating system character set identified as UTF-8.
   
   Copyright (C) 1995, 2018, Oracle and/or its affiliates. All rights reserved.
   
   
   
   GGSCI (ip-10-0-1-170.ec2.internal) 1> stats rkafka

4. Query the stock_trades table using the Athena console.

Summary

This post illustrates how you can offload reporting activity to Athena with S3 to reduce reporting costs and improve OLTP performance on the source database. This post serves as a guide for setting up a solution in the staging environment.

Deploying this solution in a production environment may require additional considerations, for example, high availability of GoldenGate hub servers, different file encoding formats for optimal query performance, and security considerations. Additionally, you can achieve similar outcomes using technologies like AWS Database Migration Service instead of GoldenGate for database CDC and Kafka Connect for the S3 sink.

About the Authors

Sreekanth Krishnavajjala is a solutions architect at Amazon Web Services.

Vinod Kataria is a senior partner solutions architect at Amazon Web Services.

Feed: MariaDB Knowledge Base Article Feed.
Author: .

With InnoDB, it is somewhat common to see the following message as an error or warning:

ERROR 1118 (42000): Row size too large (> 8126). Changing some columns to 
TEXT or BLOB may help. In current row format, BLOB prefix of 0 bytes is stored inline.

This message is raised in the following cases:

• If the innodb_strict_mode system variable is set to ON, then InnoDB will raise an error with this message if a user or application executes a DDL statement that attempts to create or alter a table and that table’s definition would allow rows that the table’s InnoDB row format can’t actually store.
• If the innodb_strict_mode system variable is set to OFF, then InnoDB will raise a warning with this message during the DDL statement instead. In this case, if a user or application actually tries to write a row that the table’s InnoDB row format can’t store, then InnoDB will raise an error with this message during the DML statement that attempts to write the row.

The root cause of the problem is that InnoDB has a maximum row size that is roughly equivalent to half of the value of the innodb_page_size system variable. See InnoDB Row Formats Overview: Maximum Row Size for more information.

InnoDB’s row formats work around this problem by storing certain kinds of variable-length columns on overflow pages. However, different row formats can store different types of data on overflow pages. Some row formats can store more data in overflow pages than others. For example, the DYNAMIC and COMPRESSED row formats can store the most data in overflow pages. To see how to determine how the various InnoDB row formats use overflow pages, see the following pages:

Checking Existing Tables for the Problem

InnoDB does not currently have an easy way to check which existing tables have this problem. See MDEV-20400 for more information.

Solving the Problem

If you encounter this error, then there are some solutions available.

Disabling InnoDB Strict Mode

The unsafe option is to disable InnoDB strict mode.

InnoDB strict mode can be disabled by setting the innodb_strict_mode system variable to OFF.

For example, even though the following table schema is too large for most InnoDB row formats to store, it can still be created when InnoDB strict mode is disabled:

SET GLOBAL innodb_default_row_format='dynamic';
SET SESSION innodb_strict_mode=OFF;
CREATE OR REPLACE TABLE tab (
   col1 varchar(40) NOT NULL,
   col2 varchar(40) NOT NULL,
   col3 varchar(40) NOT NULL,
   col4 varchar(40) NOT NULL,
   col5 varchar(40) NOT NULL,
   col6 varchar(40) NOT NULL,
   col7 varchar(40) NOT NULL,
   col8 varchar(40) NOT NULL,
   col9 varchar(40) NOT NULL,
   col10 varchar(40) NOT NULL,
   col11 varchar(40) NOT NULL,
   col12 varchar(40) NOT NULL,
   col13 varchar(40) NOT NULL,
   col14 varchar(40) NOT NULL,
   col15 varchar(40) NOT NULL,
   col16 varchar(40) NOT NULL,
   col17 varchar(40) NOT NULL,
   col18 varchar(40) NOT NULL,
   col19 varchar(40) NOT NULL,
   col20 varchar(40) NOT NULL,
   col21 varchar(40) NOT NULL,
   col22 varchar(40) NOT NULL,
   col23 varchar(40) NOT NULL,
   col24 varchar(40) NOT NULL,
   col25 varchar(40) NOT NULL,
   col26 varchar(40) NOT NULL,
   col27 varchar(40) NOT NULL,
   col28 varchar(40) NOT NULL,
   col29 varchar(40) NOT NULL,
   col30 varchar(40) NOT NULL,
   col31 varchar(40) NOT NULL,
   col32 varchar(40) NOT NULL,
   col33 varchar(40) NOT NULL,
   col34 varchar(40) NOT NULL,
   col35 varchar(40) NOT NULL,
   col36 varchar(40) NOT NULL,
   col37 varchar(40) NOT NULL,
   col38 varchar(40) NOT NULL,
   col39 varchar(40) NOT NULL,
   col40 varchar(40) NOT NULL,
   col41 varchar(40) NOT NULL,
   col42 varchar(40) NOT NULL,
   col43 varchar(40) NOT NULL,
   col44 varchar(40) NOT NULL,
   col45 varchar(40) NOT NULL,
   col46 varchar(40) NOT NULL,
   col47 varchar(40) NOT NULL,
   col48 varchar(40) NOT NULL,
   col49 varchar(40) NOT NULL,
   col50 varchar(40) NOT NULL,
   col51 varchar(40) NOT NULL,
   col52 varchar(40) NOT NULL,
   col53 varchar(40) NOT NULL,
   col54 varchar(40) NOT NULL,
   col55 varchar(40) NOT NULL,
   col56 varchar(40) NOT NULL,
   col57 varchar(40) NOT NULL,
   col58 varchar(40) NOT NULL,
   col59 varchar(40) NOT NULL,
   col60 varchar(40) NOT NULL,
   col61 varchar(40) NOT NULL,
   col62 varchar(40) NOT NULL,
   col63 varchar(40) NOT NULL,
   col64 varchar(40) NOT NULL,
   col65 varchar(40) NOT NULL,
   col66 varchar(40) NOT NULL,
   col67 varchar(40) NOT NULL,
   col68 varchar(40) NOT NULL,
   col69 varchar(40) NOT NULL,
   col70 varchar(40) NOT NULL,
   col71 varchar(40) NOT NULL,
   col72 varchar(40) NOT NULL,
   col73 varchar(40) NOT NULL,
   col74 varchar(40) NOT NULL,
   col75 varchar(40) NOT NULL,
   col76 varchar(40) NOT NULL,
   col77 varchar(40) NOT NULL,
   col78 varchar(40) NOT NULL,
   col79 varchar(40) NOT NULL,
   col80 varchar(40) NOT NULL,
   col81 varchar(40) NOT NULL,
   col82 varchar(40) NOT NULL,
   col83 varchar(40) NOT NULL,
   col84 varchar(40) NOT NULL,
   col85 varchar(40) NOT NULL,
   col86 varchar(40) NOT NULL,
   col87 varchar(40) NOT NULL,
   col88 varchar(40) NOT NULL,
   col89 varchar(40) NOT NULL,
   col90 varchar(40) NOT NULL,
   col91 varchar(40) NOT NULL,
   col92 varchar(40) NOT NULL,
   col93 varchar(40) NOT NULL,
   col94 varchar(40) NOT NULL,
   col95 varchar(40) NOT NULL,
   col96 varchar(40) NOT NULL,
   col97 varchar(40) NOT NULL,
   col98 varchar(40) NOT NULL,
   col99 varchar(40) NOT NULL,
   col100 varchar(40) NOT NULL,
   col101 varchar(40) NOT NULL,
   col102 varchar(40) NOT NULL,
   col103 varchar(40) NOT NULL,
   col104 varchar(40) NOT NULL,
   col105 varchar(40) NOT NULL,
   col106 varchar(40) NOT NULL,
   col107 varchar(40) NOT NULL,
   col108 varchar(40) NOT NULL,
   col109 varchar(40) NOT NULL,
   col110 varchar(40) NOT NULL,
   col111 varchar(40) NOT NULL,
   col112 varchar(40) NOT NULL,
   col113 varchar(40) NOT NULL,
   col114 varchar(40) NOT NULL,
   col115 varchar(40) NOT NULL,
   col116 varchar(40) NOT NULL,
   col117 varchar(40) NOT NULL,
   col118 varchar(40) NOT NULL,
   col119 varchar(40) NOT NULL,
   col120 varchar(40) NOT NULL,
   col121 varchar(40) NOT NULL,
   col122 varchar(40) NOT NULL,
   col123 varchar(40) NOT NULL,
   col124 varchar(40) NOT NULL,
   col125 varchar(40) NOT NULL,
   col126 varchar(40) NOT NULL,
   col127 varchar(40) NOT NULL,
   col128 varchar(40) NOT NULL,
   col129 varchar(40) NOT NULL,
   col130 varchar(40) NOT NULL,
   col131 varchar(40) NOT NULL,
   col132 varchar(40) NOT NULL,
   col133 varchar(40) NOT NULL,
   col134 varchar(40) NOT NULL,
   col135 varchar(40) NOT NULL,
   col136 varchar(40) NOT NULL,
   col137 varchar(40) NOT NULL,
   col138 varchar(40) NOT NULL,
   col139 varchar(40) NOT NULL,
   col140 varchar(40) NOT NULL,
   col141 varchar(40) NOT NULL,
   col142 varchar(40) NOT NULL,
   col143 varchar(40) NOT NULL,
   col144 varchar(40) NOT NULL,
   col145 varchar(40) NOT NULL,
   col146 varchar(40) NOT NULL,
   col147 varchar(40) NOT NULL,
   col148 varchar(40) NOT NULL,
   col149 varchar(40) NOT NULL,
   col150 varchar(40) NOT NULL,
   col151 varchar(40) NOT NULL,
   col152 varchar(40) NOT NULL,
   col153 varchar(40) NOT NULL,
   col154 varchar(40) NOT NULL,
   col155 varchar(40) NOT NULL,
   col156 varchar(40) NOT NULL,
   col157 varchar(40) NOT NULL,
   col158 varchar(40) NOT NULL,
   col159 varchar(40) NOT NULL,
   col160 varchar(40) NOT NULL,
   col161 varchar(40) NOT NULL,
   col162 varchar(40) NOT NULL,
   col163 varchar(40) NOT NULL,
   col164 varchar(40) NOT NULL,
   col165 varchar(40) NOT NULL,
   col166 varchar(40) NOT NULL,
   col167 varchar(40) NOT NULL,
   col168 varchar(40) NOT NULL,
   col169 varchar(40) NOT NULL,
   col170 varchar(40) NOT NULL,
   col171 varchar(40) NOT NULL,
   col172 varchar(40) NOT NULL,
   col173 varchar(40) NOT NULL,
   col174 varchar(40) NOT NULL,
   col175 varchar(40) NOT NULL,
   col176 varchar(40) NOT NULL,
   col177 varchar(40) NOT NULL,
   col178 varchar(40) NOT NULL,
   col179 varchar(40) NOT NULL,
   col180 varchar(40) NOT NULL,
   col181 varchar(40) NOT NULL,
   col182 varchar(40) NOT NULL,
   col183 varchar(40) NOT NULL,
   col184 varchar(40) NOT NULL,
   col185 varchar(40) NOT NULL,
   col186 varchar(40) NOT NULL,
   col187 varchar(40) NOT NULL,
   col188 varchar(40) NOT NULL,
   col189 varchar(40) NOT NULL,
   col190 varchar(40) NOT NULL,
   col191 varchar(40) NOT NULL,
   col192 varchar(40) NOT NULL,
   col193 varchar(40) NOT NULL,
   col194 varchar(40) NOT NULL,
   col195 varchar(40) NOT NULL,
   col196 varchar(40) NOT NULL,
   col197 varchar(40) NOT NULL,
   col198 varchar(40) NOT NULL,
   PRIMARY KEY (col1)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

But if you look closely, you’ll see that InnoDB will still raise a warning with the same message:

MariaDB [db1]> SHOW WARNINGS;
+---------+------+----------------------------------------------------------------------------------------------------------------------------------------------+
| Level | Code | Message |
+---------+------+----------------------------------------------------------------------------------------------------------------------------------------------+
| Warning | 139 | Row size too large (> 8126). Changing some columns to TEXT or BLOB may help. In current row format, BLOB prefix of 0 bytes is stored inline. |
+---------+------+----------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.000 sec)

As mentioned above, even though InnoDB is allowing the table to be created, there is still an opportunity for errors. If a user or application actually tries to write a row that the InnoDB row format can’t store, then InnoDB will raise an error during the DML statement that attempts to write the row. This creates a somewhat unsafe situation, because it means that the application has the chance to encounter an additional error while executing DML.

Converting the Table to the DYNAMIC Row Format

If the table is using either the REDUNDANT or the COMPACT row format, then a safe option is to convert it to use the DYNAMIC row format instead. If your tables were originally created on an older version of MySQL or MariaDB, then your table may be using one of these older row formats:

• In MySQL 4.1 and before, the REDUNDANT row format was the default row format.
• In MariaDB 10.1 and before, the COMPACT row format was the default row format.

The DYNAMIC row format can store more data on overflow pages than these older row formats, so converting your tables to use this row format can sometimes fix the problem.

For example:

ALTER TABLE tab ROW_FORMAT=DYNAMIC;

This workaround is a safe option, because the DYNAMIC row format may actually be able to store the data safely with this new table schema. See InnoDB DYNAMIC Row Format: Overflow Pages with the DYNAMIC Row Format for more information.

In MariaDB 10.2 and later, the DYNAMIC row format is the default row format. If your tables were originally created on one of these newer versions, then they may already be using this row format. In that case, you may need to use the next workaround.

Fitting More Columns on Overflow Pages

If the table is already using the DYNAMIC row format, then another safe option is to change the table schema, so that the row format can store more columns on overflow pages.

For VARCHAR columns, the DYNAMIC row format can only store these columns on overflow pages if the maximum length of the column is 256 bytes or longer. The original table schema is failing, because all of those small VARCHAR columns have to be stored on the row’s main data page. See InnoDB DYNAMIC Row Format: Overflow Pages with the DYNAMIC Row Format for more information.

Therefore, another safe workaround in a lot of cases is actually to increase the size of some variable-length columns, so that InnoDB’s row format can store them on overflow pages.

For example, when using InnoDB’s DYNAMIC row format and a default character set of latin1 (which requires up to 1 byte per character), the 256 byte limit means that a VARCHAR column will only be stored on overflow pages if it is at least as large as a varchar(256):

SET GLOBAL innodb_default_row_format='dynamic';
SET SESSION innodb_strict_mode=ON;
CREATE OR REPLACE TABLE tab (
   col1 varchar(256) NOT NULL,
   col2 varchar(256) NOT NULL,
   col3 varchar(256) NOT NULL,
   col4 varchar(256) NOT NULL,
   col5 varchar(256) NOT NULL,
   col6 varchar(256) NOT NULL,
   col7 varchar(256) NOT NULL,
   col8 varchar(256) NOT NULL,
   col9 varchar(256) NOT NULL,
   col10 varchar(256) NOT NULL,
   col11 varchar(256) NOT NULL,
   col12 varchar(256) NOT NULL,
   col13 varchar(256) NOT NULL,
   col14 varchar(256) NOT NULL,
   col15 varchar(256) NOT NULL,
   col16 varchar(256) NOT NULL,
   col17 varchar(256) NOT NULL,
   col18 varchar(256) NOT NULL,
   col19 varchar(256) NOT NULL,
   col20 varchar(256) NOT NULL,
   col21 varchar(256) NOT NULL,
   col22 varchar(256) NOT NULL,
   col23 varchar(256) NOT NULL,
   col24 varchar(256) NOT NULL,
   col25 varchar(256) NOT NULL,
   col26 varchar(256) NOT NULL,
   col27 varchar(256) NOT NULL,
   col28 varchar(256) NOT NULL,
   col29 varchar(256) NOT NULL,
   col30 varchar(256) NOT NULL,
   col31 varchar(256) NOT NULL,
   col32 varchar(256) NOT NULL,
   col33 varchar(256) NOT NULL,
   col34 varchar(256) NOT NULL,
   col35 varchar(256) NOT NULL,
   col36 varchar(256) NOT NULL,
   col37 varchar(256) NOT NULL,
   col38 varchar(256) NOT NULL,
   col39 varchar(256) NOT NULL,
   col40 varchar(256) NOT NULL,
   col41 varchar(256) NOT NULL,
   col42 varchar(256) NOT NULL,
   col43 varchar(256) NOT NULL,
   col44 varchar(256) NOT NULL,
   col45 varchar(256) NOT NULL,
   col46 varchar(256) NOT NULL,
   col47 varchar(256) NOT NULL,
   col48 varchar(256) NOT NULL,
   col49 varchar(256) NOT NULL,
   col50 varchar(256) NOT NULL,
   col51 varchar(256) NOT NULL,
   col52 varchar(256) NOT NULL,
   col53 varchar(256) NOT NULL,
   col54 varchar(256) NOT NULL,
   col55 varchar(256) NOT NULL,
   col56 varchar(256) NOT NULL,
   col57 varchar(256) NOT NULL,
   col58 varchar(256) NOT NULL,
   col59 varchar(256) NOT NULL,
   col60 varchar(256) NOT NULL,
   col61 varchar(256) NOT NULL,
   col62 varchar(256) NOT NULL,
   col63 varchar(256) NOT NULL,
   col64 varchar(256) NOT NULL,
   col65 varchar(256) NOT NULL,
   col66 varchar(256) NOT NULL,
   col67 varchar(256) NOT NULL,
   col68 varchar(256) NOT NULL,
   col69 varchar(256) NOT NULL,
   col70 varchar(256) NOT NULL,
   col71 varchar(256) NOT NULL,
   col72 varchar(256) NOT NULL,
   col73 varchar(256) NOT NULL,
   col74 varchar(256) NOT NULL,
   col75 varchar(256) NOT NULL,
   col76 varchar(256) NOT NULL,
   col77 varchar(256) NOT NULL,
   col78 varchar(256) NOT NULL,
   col79 varchar(256) NOT NULL,
   col80 varchar(256) NOT NULL,
   col81 varchar(256) NOT NULL,
   col82 varchar(256) NOT NULL,
   col83 varchar(256) NOT NULL,
   col84 varchar(256) NOT NULL,
   col85 varchar(256) NOT NULL,
   col86 varchar(256) NOT NULL,
   col87 varchar(256) NOT NULL,
   col88 varchar(256) NOT NULL,
   col89 varchar(256) NOT NULL,
   col90 varchar(256) NOT NULL,
   col91 varchar(256) NOT NULL,
   col92 varchar(256) NOT NULL,
   col93 varchar(256) NOT NULL,
   col94 varchar(256) NOT NULL,
   col95 varchar(256) NOT NULL,
   col96 varchar(256) NOT NULL,
   col97 varchar(256) NOT NULL,
   col98 varchar(256) NOT NULL,
   col99 varchar(256) NOT NULL,
   col100 varchar(256) NOT NULL,
   col101 varchar(256) NOT NULL,
   col102 varchar(256) NOT NULL,
   col103 varchar(256) NOT NULL,
   col104 varchar(256) NOT NULL,
   col105 varchar(256) NOT NULL,
   col106 varchar(256) NOT NULL,
   col107 varchar(256) NOT NULL,
   col108 varchar(256) NOT NULL,
   col109 varchar(256) NOT NULL,
   col110 varchar(256) NOT NULL,
   col111 varchar(256) NOT NULL,
   col112 varchar(256) NOT NULL,
   col113 varchar(256) NOT NULL,
   col114 varchar(256) NOT NULL,
   col115 varchar(256) NOT NULL,
   col116 varchar(256) NOT NULL,
   col117 varchar(256) NOT NULL,
   col118 varchar(256) NOT NULL,
   col119 varchar(256) NOT NULL,
   col120 varchar(256) NOT NULL,
   col121 varchar(256) NOT NULL,
   col122 varchar(256) NOT NULL,
   col123 varchar(256) NOT NULL,
   col124 varchar(256) NOT NULL,
   col125 varchar(256) NOT NULL,
   col126 varchar(256) NOT NULL,
   col127 varchar(256) NOT NULL,
   col128 varchar(256) NOT NULL,
   col129 varchar(256) NOT NULL,
   col130 varchar(256) NOT NULL,
   col131 varchar(256) NOT NULL,
   col132 varchar(256) NOT NULL,
   col133 varchar(256) NOT NULL,
   col134 varchar(256) NOT NULL,
   col135 varchar(256) NOT NULL,
   col136 varchar(256) NOT NULL,
   col137 varchar(256) NOT NULL,
   col138 varchar(256) NOT NULL,
   col139 varchar(256) NOT NULL,
   col140 varchar(256) NOT NULL,
   col141 varchar(256) NOT NULL,
   col142 varchar(256) NOT NULL,
   col143 varchar(256) NOT NULL,
   col144 varchar(256) NOT NULL,
   col145 varchar(256) NOT NULL,
   col146 varchar(256) NOT NULL,
   col147 varchar(256) NOT NULL,
   col148 varchar(256) NOT NULL,
   col149 varchar(256) NOT NULL,
   col150 varchar(256) NOT NULL,
   col151 varchar(256) NOT NULL,
   col152 varchar(256) NOT NULL,
   col153 varchar(256) NOT NULL,
   col154 varchar(256) NOT NULL,
   col155 varchar(256) NOT NULL,
   col156 varchar(256) NOT NULL,
   col157 varchar(256) NOT NULL,
   col158 varchar(256) NOT NULL,
   col159 varchar(256) NOT NULL,
   col160 varchar(256) NOT NULL,
   col161 varchar(256) NOT NULL,
   col162 varchar(256) NOT NULL,
   col163 varchar(256) NOT NULL,
   col164 varchar(256) NOT NULL,
   col165 varchar(256) NOT NULL,
   col166 varchar(256) NOT NULL,
   col167 varchar(256) NOT NULL,
   col168 varchar(256) NOT NULL,
   col169 varchar(256) NOT NULL,
   col170 varchar(256) NOT NULL,
   col171 varchar(256) NOT NULL,
   col172 varchar(256) NOT NULL,
   col173 varchar(256) NOT NULL,
   col174 varchar(256) NOT NULL,
   col175 varchar(256) NOT NULL,
   col176 varchar(256) NOT NULL,
   col177 varchar(256) NOT NULL,
   col178 varchar(256) NOT NULL,
   col179 varchar(256) NOT NULL,
   col180 varchar(256) NOT NULL,
   col181 varchar(256) NOT NULL,
   col182 varchar(256) NOT NULL,
   col183 varchar(256) NOT NULL,
   col184 varchar(256) NOT NULL,
   col185 varchar(256) NOT NULL,
   col186 varchar(256) NOT NULL,
   col187 varchar(256) NOT NULL,
   col188 varchar(256) NOT NULL,
   col189 varchar(256) NOT NULL,
   col190 varchar(256) NOT NULL,
   col191 varchar(256) NOT NULL,
   col192 varchar(256) NOT NULL,
   col193 varchar(256) NOT NULL,
   col194 varchar(256) NOT NULL,
   col195 varchar(256) NOT NULL,
   col196 varchar(256) NOT NULL,
   col197 varchar(256) NOT NULL,
   col198 varchar(256) NOT NULL,
   PRIMARY KEY (col1)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;

And when using InnoDB’s DYNAMIC row format and a default character set of utf8 (which requires up to 3 bytes per character), the 256 byte limit means that a VARCHAR column will only be stored on overflow pages if it is at least as large as a varchar(86):

SET GLOBAL innodb_default_row_format='dynamic';
SET SESSION innodb_strict_mode=ON;
CREATE OR REPLACE TABLE tab (
   col1 varchar(86) NOT NULL,
   col2 varchar(86) NOT NULL,
   col3 varchar(86) NOT NULL,
   col4 varchar(86) NOT NULL,
   col5 varchar(86) NOT NULL,
   col6 varchar(86) NOT NULL,
   col7 varchar(86) NOT NULL,
   col8 varchar(86) NOT NULL,
   col9 varchar(86) NOT NULL,
   col10 varchar(86) NOT NULL,
   col11 varchar(86) NOT NULL,
   col12 varchar(86) NOT NULL,
   col13 varchar(86) NOT NULL,
   col14 varchar(86) NOT NULL,
   col15 varchar(86) NOT NULL,
   col16 varchar(86) NOT NULL,
   col17 varchar(86) NOT NULL,
   col18 varchar(86) NOT NULL,
   col19 varchar(86) NOT NULL,
   col20 varchar(86) NOT NULL,
   col21 varchar(86) NOT NULL,
   col22 varchar(86) NOT NULL,
   col23 varchar(86) NOT NULL,
   col24 varchar(86) NOT NULL,
   col25 varchar(86) NOT NULL,
   col26 varchar(86) NOT NULL,
   col27 varchar(86) NOT NULL,
   col28 varchar(86) NOT NULL,
   col29 varchar(86) NOT NULL,
   col30 varchar(86) NOT NULL,
   col31 varchar(86) NOT NULL,
   col32 varchar(86) NOT NULL,
   col33 varchar(86) NOT NULL,
   col34 varchar(86) NOT NULL,
   col35 varchar(86) NOT NULL,
   col36 varchar(86) NOT NULL,
   col37 varchar(86) NOT NULL,
   col38 varchar(86) NOT NULL,
   col39 varchar(86) NOT NULL,
   col40 varchar(86) NOT NULL,
   col41 varchar(86) NOT NULL,
   col42 varchar(86) NOT NULL,
   col43 varchar(86) NOT NULL,
   col44 varchar(86) NOT NULL,
   col45 varchar(86) NOT NULL,
   col46 varchar(86) NOT NULL,
   col47 varchar(86) NOT NULL,
   col48 varchar(86) NOT NULL,
   col49 varchar(86) NOT NULL,
   col50 varchar(86) NOT NULL,
   col51 varchar(86) NOT NULL,
   col52 varchar(86) NOT NULL,
   col53 varchar(86) NOT NULL,
   col54 varchar(86) NOT NULL,
   col55 varchar(86) NOT NULL,
   col56 varchar(86) NOT NULL,
   col57 varchar(86) NOT NULL,
   col58 varchar(86) NOT NULL,
   col59 varchar(86) NOT NULL,
   col60 varchar(86) NOT NULL,
   col61 varchar(86) NOT NULL,
   col62 varchar(86) NOT NULL,
   col63 varchar(86) NOT NULL,
   col64 varchar(86) NOT NULL,
   col65 varchar(86) NOT NULL,
   col66 varchar(86) NOT NULL,
   col67 varchar(86) NOT NULL,
   col68 varchar(86) NOT NULL,
   col69 varchar(86) NOT NULL,
   col70 varchar(86) NOT NULL,
   col71 varchar(86) NOT NULL,
   col72 varchar(86) NOT NULL,
   col73 varchar(86) NOT NULL,
   col74 varchar(86) NOT NULL,
   col75 varchar(86) NOT NULL,
   col76 varchar(86) NOT NULL,
   col77 varchar(86) NOT NULL,
   col78 varchar(86) NOT NULL,
   col79 varchar(86) NOT NULL,
   col80 varchar(86) NOT NULL,
   col81 varchar(86) NOT NULL,
   col82 varchar(86) NOT NULL,
   col83 varchar(86) NOT NULL,
   col84 varchar(86) NOT NULL,
   col85 varchar(86) NOT NULL,
   col86 varchar(86) NOT NULL,
   col87 varchar(86) NOT NULL,
   col88 varchar(86) NOT NULL,
   col89 varchar(86) NOT NULL,
   col90 varchar(86) NOT NULL,
   col91 varchar(86) NOT NULL,
   col92 varchar(86) NOT NULL,
   col93 varchar(86) NOT NULL,
   col94 varchar(86) NOT NULL,
   col95 varchar(86) NOT NULL,
   col96 varchar(86) NOT NULL,
   col97 varchar(86) NOT NULL,
   col98 varchar(86) NOT NULL,
   col99 varchar(86) NOT NULL,
   col100 varchar(86) NOT NULL,
   col101 varchar(86) NOT NULL,
   col102 varchar(86) NOT NULL,
   col103 varchar(86) NOT NULL,
   col104 varchar(86) NOT NULL,
   col105 varchar(86) NOT NULL,
   col106 varchar(86) NOT NULL,
   col107 varchar(86) NOT NULL,
   col108 varchar(86) NOT NULL,
   col109 varchar(86) NOT NULL,
   col110 varchar(86) NOT NULL,
   col111 varchar(86) NOT NULL,
   col112 varchar(86) NOT NULL,
   col113 varchar(86) NOT NULL,
   col114 varchar(86) NOT NULL,
   col115 varchar(86) NOT NULL,
   col116 varchar(86) NOT NULL,
   col117 varchar(86) NOT NULL,
   col118 varchar(86) NOT NULL,
   col119 varchar(86) NOT NULL,
   col120 varchar(86) NOT NULL,
   col121 varchar(86) NOT NULL,
   col122 varchar(86) NOT NULL,
   col123 varchar(86) NOT NULL,
   col124 varchar(86) NOT NULL,
   col125 varchar(86) NOT NULL,
   col126 varchar(86) NOT NULL,
   col127 varchar(86) NOT NULL,
   col128 varchar(86) NOT NULL,
   col129 varchar(86) NOT NULL,
   col130 varchar(86) NOT NULL,
   col131 varchar(86) NOT NULL,
   col132 varchar(86) NOT NULL,
   col133 varchar(86) NOT NULL,
   col134 varchar(86) NOT NULL,
   col135 varchar(86) NOT NULL,
   col136 varchar(86) NOT NULL,
   col137 varchar(86) NOT NULL,
   col138 varchar(86) NOT NULL,
   col139 varchar(86) NOT NULL,
   col140 varchar(86) NOT NULL,
   col141 varchar(86) NOT NULL,
   col142 varchar(86) NOT NULL,
   col143 varchar(86) NOT NULL,
   col144 varchar(86) NOT NULL,
   col145 varchar(86) NOT NULL,
   col146 varchar(86) NOT NULL,
   col147 varchar(86) NOT NULL,
   col148 varchar(86) NOT NULL,
   col149 varchar(86) NOT NULL,
   col150 varchar(86) NOT NULL,
   col151 varchar(86) NOT NULL,
   col152 varchar(86) NOT NULL,
   col153 varchar(86) NOT NULL,
   col154 varchar(86) NOT NULL,
   col155 varchar(86) NOT NULL,
   col156 varchar(86) NOT NULL,
   col157 varchar(86) NOT NULL,
   col158 varchar(86) NOT NULL,
   col159 varchar(86) NOT NULL,
   col160 varchar(86) NOT NULL,
   col161 varchar(86) NOT NULL,
   col162 varchar(86) NOT NULL,
   col163 varchar(86) NOT NULL,
   col164 varchar(86) NOT NULL,
   col165 varchar(86) NOT NULL,
   col166 varchar(86) NOT NULL,
   col167 varchar(86) NOT NULL,
   col168 varchar(86) NOT NULL,
   col169 varchar(86) NOT NULL,
   col170 varchar(86) NOT NULL,
   col171 varchar(86) NOT NULL,
   col172 varchar(86) NOT NULL,
   col173 varchar(86) NOT NULL,
   col174 varchar(86) NOT NULL,
   col175 varchar(86) NOT NULL,
   col176 varchar(86) NOT NULL,
   col177 varchar(86) NOT NULL,
   col178 varchar(86) NOT NULL,
   col179 varchar(86) NOT NULL,
   col180 varchar(86) NOT NULL,
   col181 varchar(86) NOT NULL,
   col182 varchar(86) NOT NULL,
   col183 varchar(86) NOT NULL,
   col184 varchar(86) NOT NULL,
   col185 varchar(86) NOT NULL,
   col186 varchar(86) NOT NULL,
   col187 varchar(86) NOT NULL,
   col188 varchar(86) NOT NULL,
   col189 varchar(86) NOT NULL,
   col190 varchar(86) NOT NULL,
   col191 varchar(86) NOT NULL,
   col192 varchar(86) NOT NULL,
   col193 varchar(86) NOT NULL,
   col194 varchar(86) NOT NULL,
   col195 varchar(86) NOT NULL,
   col196 varchar(86) NOT NULL,
   col197 varchar(86) NOT NULL,
   col198 varchar(86) NOT NULL,
   PRIMARY KEY (col1)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;