When you want to process your application logs, Elasticsearch bubbles up as the go-to technology. Elasticsearch is a popular, open source distributed search and analytics package. The Elasticsearch stack adds a usability layer to search, analyze and process your Apache Lucene data. Open Distro for Elasticsearch is a full open source package designed by AWS to enhance and protect the open source capabilities of the base Elasticsearch engine. This distribution bundles critical open source components including security, cluster diagnostics, alerting and SQL capabilities for Elasticsearch. My talk will guide you through Open Distro features and build tools. I will also cover the project's community driven approach to building a great open source search stack where you can join in and collaborate.
It's easy for modern, distributed, high-scale applications to hide database performance and efficiency problems. Optimizing performance of such complex systems at scale requires some skill, but more importantly, it requires a sound strategy and good observability, because you can't optimize what you can't measure. This session explains a performance measurement and optimization process anyone can use to deliver results predictably, optimizing customer experience while freeing up computing resources and saving money.
The session begins with what to measure and how; how to analyze it; how to categorize problems into one of three types; and three matching strategies to use in optimization as a result. It is a recursive method that can be used at any scale, from a data center with many types of databases cooperating as one, to a single server and drilling down to a single query. Along the way, we'll discuss related concepts such as internally- and externally-focused golden signals of performance and resource sufficiency, workload quality of service, and more.
PMM 2.0 represents a significant advance in terms of monitoring for Open Source Databases. Come to this session in order to learn about the following:
* Query Analytics improvements
* Architectural changes
* API and GUI configuration
This talk will focus on the self-managed nature of Uber's database monitoring and how we've leveraged our open source time series database M3DB to support massive multi-region scale and high cardinality monitoring.
We'll cover how we monitor applications, databases, and their interactions and how we automatically setup application specific dashboards and alerts. This includes things like the ability to alert on metrics like P99 latency and slow queries for a given application at the per-table and per-query level.
Of course, automated and fine grained monitoring requires the ability to ingest, persist, and query massive amounts of high cardinality time series data. We'll talk about the architecture of M3DB and how we've leveraged it at Uber to scale our monitoring systems to billion of unique time series and 10s of millions of data points per second.
We'll conclude the talk with an overview of our Prometheus and Kubernetes integrations, explaining how you can start leveraging M3DB for your own workloads. Finally, we'll give a brief overview of our plans to evolve M3DB into a general purpose, horizontally scalable event store.
GTIDs were introduced to solve replication problems and improve database consistency.
When, accidentally, transactions occur on a replica, this introduces GTIDs on that replica that don't exist on the master. When, on a master failover, this replica becomes the new master, and the corresponding binlogs of the errant GTIDs are already purged, replication breaks on the replicas of this new master, because those missing GTIDs can't be retrieved from the binlogs of this new master.
This presentation will talk about GTIDs and how to detect errant GTIDs on a replica (before the corresponding binlogs are purged) and how to look at the corresponding transactions in the binlogs. I'll give some examples of transactions that could happen on a replica that didn't originate from a primary node, explain how this is possible and share some tips on how to avoid this.
Basic understanding of MySQL database replication is assumed.
Have you struggled with identifying issues in MySQL?
Come listen to how Verisure experienced an issue that was identified and resolved using PMM (Percona Monitoring and Management). Verisure was able to find the offending query/tuning parameter, make modifications, and then observe the impact over time thanks to PMM.
The client stack provides the data plane required to access MySQL from client applications. At Facebook we have a fully featured client stack along with MySQL protocol proxies that we use to scale MySQL across regions. This talk will provide an overview of the MySQL client libraries in use at Facebook, our MySQL protocol proxies, and our integrations with internal service discovery, logging, and monitoring systems.
Optimizing MySQL performance and troubleshooting MySQL problems are two of the most critical and challenging tasks for MySQL DBA's. The databases powering your applications need to be able to handle heavy traffic loads while remaining responsive and stable so that you can deliver an excellent user experience. Further, DBA's are also expected to find cost-efficient means of solving these issues.In this presentation, we will discuss how you can optimize and troubleshoot MySQL performance and demonstrate how Percona Monitoring and Management (PMM) enables you to solve these challenges using free and open source software. We will look at specific, common MySQL problems and review the essential components in PMM that allow you to diagnose and resolve them.
Loki is a horizontally-scalable, highly-available log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate, as it does not index the contents of the logs, but rather labels for each log stream.
Loki initially targets Kubernetes logging, using Prometheus service discovery to gather labels and metadata about log streams. By using the same index and labels as Prometheus, Loki enables you to easily switch between metrics and logs, enhancing observability and streamlining the incident response process â€“ a workflow we have built into the latest version of Grafana.
In this talk, we will discuss the motivation behind Loki, its design and architecture, and what the future holds. It's early days, but so far the response to the project has been overwhelming, with more the 4.5k GitHub stars and over 12hrs at the top spot on Hacker News.
Loki is an opensource project, Apache licensed.
Do you already run stock PMM in your environment and want to learn how you extend the PMM platform? Come learn about:
1. Dashboard Customizations
How to create custom dashboard from existing graphs, or build Cross Server Dashboards
2. External Exporters - Monitor any service, anywhere!
Adding an exporter, view the data in data exploration, to deploying a working Dashboard
3. Working with custom queries (MySQL and PostgreSQL)
Execute SELECT statements against your database and store in Prometheus
Build Dashboards relevant to your environment
4. Customizing Exporter Options
Enable de-activated functionality that applies to your environment
5. Using Grafana Alerting
How to set up channels (SMTP, Slack, etc)
How to configure thresholds and alerts
6. Using MySQL/PostgreSQL Data Source
Execute SELECT statements against your database and plot your application metrics